vibecert

tl;dr I wrote a Claude Code skill, VibeCert, to rate the vibe coding process. It is designed to encourage me to always use generative tools sensibly, and to show to others that I have done so. The skill is available here.

A post last year reflected on good practices for using generative coding tools. It commented that the presence of these practices might also be used as the basis for judging whether these tools have been used correctly. Since this post, the use of generative coding tools has, at least in the outputs I see, increased significantly, and I myself now use them to generate more and more software features. While I don’t see the rise of these tools as an immediate threat to jobs – instead, I fully subscribe to the idea that ‘ChatGPT is my Toothpaste Robot’ – I do think that, if used incorrectly, technical debt is going to be a significant problem in the future, especially if there are ever shifts in the way generative tools operate or their availability. At the same time, it wouldn’t seem sensible to fall behind by abandoning these tools altogether.

I am therefore prompted to revisit what it means to use generative coding tools correctly, and to formalise these ideas into a tangible meta tool that can provide me with a methodological quality score. Attaching this score to code will, ideally, force me to always use these automated software development tools in a sustainable way, and also show to others that I am not carelessly injecting slop into my outputs. I call this tool VibeCert (a certification for vibe coded outputs), and the details of its construction are provided below.

Assessing vibe coding practices

Before actually creating the tool itself, we can define general metrics for assessing vibe coding practices, particularly in respect of limiting technical debt:

1. Follow-up questions. As noted in last year’s post, I still see asking follow-up questions about output as a really important indicator of methodological quality. A record of questions being asked such as “What does this particular new piece of Python syntax that I haven’t seen before do?”, “Why did you choose to structure Line 41 in this way?” and “Is this really the conventional way to produce this value?”, although evidence of slightly anthropomorphising the model, is, I believe, a good way to gauge whether understanding has been achieved at best, and at the very least a way to identify whether outputs have been reviewed. Naturally assessing the granularity and relevance of questions is important.

2. Coding guidelines. Flipping our approach round to assess not what knowledge an individual has tried to extract from the outputs but what knowledge they have injected into them, we can also ask how well the tool is emulating our own (hopefully high-quality) non-generative coding practices. To do this, we can measure a generative model’s use of our own coding guidelines in its output. In the case of Claude Code this will be the use of rules, or the review of Claude.md files (‘memory’), for example.

3. Output amendments. An evolution of metric (1) is not just determining how many clarifying questions have been asked, but how many actual edits have been made directly (potentially as a result of those questions); not accepting outputs verbatim but, either via a dialogue with the model or via direct edits, altering outputs. While this may, of course, introduce less-optimal code, it is at least an indicator of the first two metrics: that some understanding has been gained (or the code has been reviewed), and that one’s own coding style has been injected into the output. Both of these reduce technical debt.

4. Prompt granularity. Our assessment thus far has focussed on outputs, but assessing inputs (prompts) is equally as important. Asking “build me a React application to deliver an e-commerce site”, for example, naturally invites technical debt in a manner that asking for more granular, iterative development does not, such as “build me the connection string of a database connection class” followed by requests for other supporting features.

5. Secondary tool usage. The next indicator of methodological quality should be the extent to which the model has used, or been prompted to use, traditional code analysis tools during the construction of the output. Has it (implemented and) run unit tests against the output? Has it (installed and) run a linter against the output? Has it iterated based on the above? Having automated checks used as part of the generative process, and/or having them present with good coverage of the generated code, suggests proper usage.

6. Generative comments. Finally, and perhaps counterintuitively, the number of generative comments should be used as an inverse indicator of correct usage. I believe that having a generative tool explain itself in comments is often used as a proxy for understanding – I’ll just read these in the future if I ever need to understand what the code is doing – mostly because I have found myself tempted to do this exact thing. But I think a complex idea is still complex whether conveyed in prose or in code.

Weighting good-practice scores

There are some situations where higher good-practice scores are more important – or, conversely, where lower scores are more acceptable – and this should be factored into any final qualitative assessment of vibe coding practices:

• Quantity of generative output. The first natural thing to determine is how much of the output is actually generative. Metrics should only be applied to AI output.

• Quantity of output. More broadly, the general length of the generated output should be considered. This can impact the meaningfulness of metrics, particularly (1) where an output may be so short that there genuinely isn’t anything to query.

• Code criticality. How central the generated code is to the piece of software under development as a whole should be considered. A database connection class is, broadly, more critical than a utility class designed to support admin-initiated manual backups of that same database. Thus, in the former case, good practices are even more important.

• Code risk. If the generative code is designed, for example, to handle sensitive data, or call lots of external services, we can consider this code to be high risk. As with code criticality, good practice checks are thus of high importance. In identifying high-risk code, one also builds into the process for free a check that the model hasn’t emitted any harmful outputs.

Deriving good-practice scores and weights

How do we derive the metrics, and weight them, as described above? Well, at the risk of entering a chicken and egg situation, most can be measured by AI itself. Assuming VibeCert is applied to Git branches, we can present the code, the diff, and the chat history related to the branch to generative AI, and have it score relevant metrics from above¹. We can provide the model with a scale, from 1 to 3, to help with this scoring. For example, for metric (1): 1 - No questions were asked about the code; 2 - Questions were asked about the code, but they were very broad or irrelevant; 3 - Questions were asked about the code, and they were very specific. Non-generative (deterministic) techniques can be used in parallel to derive metrics that don’t require AI ‘judgement’.

We can then have the model also help derive our weights. For example, we can ask for an assessment on whether the length of the generated output is considered short, normal or long, and ascribe a score to this of 1, 2 or 3, respectively. Other weights may necessitate support from non-generative measures such as counting lines in the codebase, or calling complementary python code to, for example, identify high-risk code that makes external connections.

With our good-practice scores and our weights, and assuming we always score and weight 1 - 3, we can then produce a lookup table with final scores. Continuing our running example, in such a table (1,1) (No questions; short document) might result in a score of 2 (no questions is not ideal, but the document is short, so provide a middle score), while (1, 3) (No questions; long document) might result in a score of 1 (no questions is not ideal, and the document is long, so provide the lowest score).

Creating a Claude Code skill to realise the tool

To realise the VibeCert logic described thus far, a Claude Code skill was implemented following the specification. The skill can be viewed in this repository. To be somewhat meta, Claude itself was asked to implement the skill by passing the text of this post so far to it. This also provided an opportunity to see the skill in practice (described in the following section), by using the skill to score the production of itself (!).

Key features of the skill are:

• A detailed rubric conveying the information described in this post to the model, completing the skill.

• Mixture of generative (i.e. prompts) and non-generative (i.e. utility classes in a python package) metric evidence.

• Detailed formatted assessment output, containing evidence, weights and explanations.

• Support for publishing the output of the assessment to a public repository, which can then be referenced from a suitable location (commit message, PR description, etc).

Worked example

Using the VibeCert skill to assess the creation of itself yielded the following output: https://vibecert.github.io/bcf81d7/. The assessment is a fair reflection of my practice in creating the skill across most areas: scoring high in follow-up questions (I asked a lot), secondary tool usage (many tests were added), and generative comment absence (I rarely ask for these) is accurate. Scoring low in amendments and guidelines is also accurate, as I observed myself asking for less and less changes, and making less and less changes (to align with my own guidelines), as time went on. Although, I would probably dispute the impact that (the generative output) not following a single guideline (direct name imports) had on the whole score. Equally, prompt granularity was good in practice, but it seems like it just crept over an acceptable threshold and reduced the overall score; an issue when working with the limited range of 1 - 3.

Caveats

There are obviously several caveats to the use of this tool:

• The tool is Claude (Code) specific, but the good-practice metrics can be generalised.

• It very much represents an honesty box situation. There is nothing to stop people fabricating chat logs, for example, maybe even generatively. However, one would hope that in doing so they would reflect at least to a degree on their outputs, in which case the tool has served its purpose.

• As a hangover from my academic days where my mantra was look hard enough and you’ll find someone else working on the same problem, and talk yourself out of doing any work in the process, so just push forward, I haven’t looked into the current state of vibe coding assessment tools.

• And probably the biggest elephant in the room is that no amount of reviewing generative code output can truly negate technical debt compared to writing that code oneself. Nor can it negate potentially deskilling at the same time. For these reasons, I remain actively critical of vibe code in my workflows.