In the world of web development, turning a design into a functional website usually involves a lot of coding and careful testing. What if we simplified this process and made it easier and faster to convert web designs to work websites? WebSight is a new dataset intended to build AI systems that can convert screenshots into HTML code.
Challenge
Turning website designs or screenshots into HTML code usually requires an experienced developer. But what if this could be more efficient? Motivated by this question, we investigated how to use vision language models (VLMs) in web development to create low-coded solutions that improve efficiency.
Today, the main challenge to that goal is the lack of high quality datasets tailored for this task. WebSight aims to fill that gap.
WebSight: Large synthetic dataset of screenshot/HTML code pairs
In January 2024, we introduced WebSight-V0.1, a synthetic dataset consisting of 823,000 pairs of HTML code and corresponding screenshots. This dataset is designed to train AI models to process and transform visual web design into functional HTML code. By focusing on synthetic data, we bypassed the noise and complexity that are common in real HTML, allowing AI models to learn efficiently.
In addition to community feedback, we updated our dataset to WebSight-V0.2 following our initial release and construction, and introduced significant improvements. These extensions feature switching to Tailwind CSS instead of traditional CSS, using real images in screenshots. Additionally, we have expanded our dataset to 2 million examples.
Sightseer: A fine-tuned model with WebSight
Using the WebSight dataset, we’ve fine-tuned future Foundation Vision-Language models to get Sightseer, a model that can convert web page screenshots into functional HTML code. Stightser further demonstrates the ability to incorporate images into generated HTML that are very similar to those in the original screenshot.
Towards more powerful tools unlocked by visual language models
By repeating WebSight, our goal is to build a more capable AI system that simplifies the process of turning UI design into functional code. This reduces developer iteration times by quickly converting paper UI sketches into functional code, making this process more accessible to non-developers. This is one of many real applications of visual language models. With Open-Source Websight, the community encourages us to work with us to build more powerful tools for UI development.
resource
