tried the latest and greatest "AI" against our open issues, "fable" that's supposed to be the end of programming forever. if i am being extremely generous, on first glance, 1/5. on review, 0/5. attempted issues were extremely easy, i think with reading the library these would take about 15 minutes each, tops. These are issues written for other human beings and are relatively sparse on details, in this case a quiet project with me and another person as main authors, but on review, these should all be doable with a brief skim of the code and docs - a human could parse the details here and implement these with minimal discussion or time. I timed myself, 5-15m each, as the primary author.
- 147 - everything in our package is run by events, so nodes expiring should be an event. the "AI" took the issue literally and just suppressed errors in 2/3 runners. incomplete, undesireable. fail.
- 239 - no output was emitted, claimed issue was solved, fail. This one in particular is minimal and easy, a three line fix.
- 200 - closest to success, initially ruled success, but on further review failed. Need to turn a scalar-valued reference into a set. This has the literal solution in the issue text. "AI" implementation increased algorithmic complexity of O(1) to O(n). unacceptable, fail on review.
- 110 - handled in the laziest way imaginable that latched on to the issue title but failed to read the actual problem within the issue text, which is relatively verbose and poses several possible solutions. "AI" chose none, did the wrong thing. fail.
- 166 - this one actually was already solved and i wanted to see if the "AI" recognized that. it added an additional unnecessary mechanism that made the relevant code less efficient. fail.
model took ~60 minutes, used my entire token allotment, roughly the time it took me to do all of them correctly. i'd say it would take someone unfamiliar with the library 2x that but that's non-empirical, just a guess. if i had to sit and coach the "AI" through all of these it would take longer, i'm sure, even assuming i had unlimited tokens. easy issues. each of them is a few lines. all it takes is being able to understand the context of what's being done.
Mastodon Feed
