This is part of a two-part series about our experience making a ChatGPT plugin at Dart. In addition to this post, check out the one about How to make a great ChatGPT plugin.
What I learned from making a top ChatGPT plugin
I was lucky enough to get very early access to be a ChatGPT plugin developer because our product is very AI-oriented, and I am in close touch with OpenAI. This post is about what we learned from making and launching the plugin.
We released Dart publicly a couple of months ago and launched the ChatGPT plugin that allows users to interact with Dart and manage their products directly from ChatGPT a few weeks ago.
Developing the plugin has made several things clear. For instance, ChatGPT has rocketed to success so fast that the company around it (OpenAI) is still playing catch-up and is quite disorganized. Its users love it so much that they are accustomed to a pretty bad UX, which is all that OpenAI has been able to deliver on several levels. Unlike the chatbot itself, the plugins portion of the product may not yet have product-market fit, although our plugin is moderately successful, so all is not lost.
We get a slow and steady flow of signups from ChatGPT
Since launching the app in the public plugin store, we have had a steady stream of registrations from ChatGPT and signups for Dart.
These signups have tapered off slowly, but a deep pool of ChatGPT users are interested in the Dart plugin. We expect that if OpenAI continues to invest in plugins (e.g., making search/discovery better) and continues to roll them out to more users, we will have an essentially limitless pool of potential conversions because there are so many ChatGPT users.
Our retention of users that come from ChatGPT is lower than our retention of standard users (as many people were just checking things out without a real intention to use), but it's not too bad.
OpenAI is the biggest early-stage startup ever
Early-stage startups are often characterized by a lack of organization, a free-wheeling nature, a lack of documentation and process, and an absence of well-defined roles for individual employees. OpenAI checks all of these boxes and more, even though it has happened to develop the fastest-growing consumer app ever.
Fundamentally, hitting PMF (product-market fit) with ChatGPT hasn’t made the company, its processes, or its documentation any more developed than they would be otherwise. Although I am sure they are doing their best, this has ramifications for how easy they are to work with. For instance, the only way for me to get a review of the Dart ChatGPT plugin was to bug anyone and everyone at OpenAI I could get in touch with, including through my support rep, various Slack and WhatsApp groups, the development forum, the official support website and contact methods, and even through personal relationships at OpenAI.
Another example of it still being the Wild West over at OpenAI HQ is how scattered their documentation is. I identified at least six places where plugin requirements were documented (see this post for a list). None of the six pages were consistent; some were directly contradictory. The kicker was that there were also secret undocumented requirements that I was informed of by OpenAI support during the review process.
I believe that OpenAI will gradually get its act together (as most massively successful early-stage startups eventually do), and I have a lot of sympathy for the growing pains they’re going through. Despite this, their products are likely a bit more challenging to work with now than they would be if they came from a more mature organization.
ChatGPT plugins may not have product-market fit (PMF) yet
Raza Habib, the CEO of Humanloop, sat down with Sam Altman, the CEO of OpenAI, and learned a lot about the company’s roadmap. I’m in direct touch with Raza, but he summed up the situation in a blog post that has since been removed but is still accessible here.
Plugins “don’t have PMF” and are probably not coming to the API anytime soon
A lot of developers are interested in getting access to ChatGPT plugins via the API but Sam said he didn’t think they’d be released any time soon. The usage of plugins, other than browsing, suggests that they don’t have PMF yet. He suggested that a lot of people thought they wanted their apps to be inside ChatGPT but what they really wanted was ChatGPT in their apps.
However, this story gets more complicated. Since this meeting, Sam has essentially retracted these statements. In some groups I’m in, employees of OpenAI have said that they are still moving forward with rolling out plugins and considering API access. Since Sam’s notes, OpenAI has also broadened access to plugins in a few ways, which they probably wouldn’t be doing if it were dead in the water.
Many people seem to be using and reusing the Dart plugin, so I’m not convinced that plugins won’t work. They may need more polish to get popular, though—the UI has suffered some and could use more work.
If OpenAI invests in them, plugins will get to PMF if they aren’t there already.
ChatGPT users are accustomed to bad UX now
One abundantly clear thing is that, for better or worse, ChatGPT users are willing to put up with a lackluster UX at this point because they are so happy with the core product. For early products, being 10x better than what came before will mean that users are willing to overlook the fact that the rest of the experience is subpar.
The biggest thing that stands out in this respect is the very slow loading times. The traditional rule that web interactions must always be measured in milliseconds has gone out of the window because every interaction with ChatGPT takes forever. We have noticed that users’ willingness to forgive has even transferred to the Dart plugin. When beta users had interactions that took a long time due to latency or retries on Dart's end, they were more willing to forgive these because everything with ChatGPT takes a long time. We have ironed out these issues on our end, but ChatGPT has improved their end at a glacial pace by traditional standards.
Furthermore, many of the UI and UX interactions with ChatGPT, particularly with its plugins, still feel rough and raw (almost as if they were made very quickly, perhaps with much of the code written by an AI). Popups, hover states, etc., are all clearly less polished than they are in most apps, and users don’t seem to care.
We expect the OpenAI team to polish most of this over time, but seeing such a clear example of a 10x feature leading users to forgive other glaring issues is pretty cool!
The future of AI
At Dart, we have been building the future of task and product management with AI for a while. Integrating with ChatGPT was an exciting next step in this process. It’s a natural fit because ChatGPT is the future of AI-based chat, and Dart is the future of AI-based project management.
If you’re interested in learning more, check out this post about how to go about actually making a plugin like ours in practice. Or check out Dart here.
