AI Prompt Engineering Progression

Introduction

AI is becoming increasingly important in workflows from generative text to generative images. This exercise explores how prompts affect the images that are generated by the model. For this experiment, I will start with a non-specific prompt and add detail for each iteration of the initial prompt. The goal is to test the limits of the AI’s functions and identify the strengths and weaknesses of the image outputs for professional workflows.

Tool Choice

I chose Adobe Express because I thought that it would integrate well with the rest of the Adobe Creative Suite. I also assume that the AI generation model used is the same model used in the other applications which means that the prompting techniques and output results I learn with Adobe Express will produce similar results across other products.

Adobe Express also had the lowest barrier to entry. It clearly advertised
that it was free and it was easy to get to the UI interface. I had heard of
DALL-E before, so I tried to use that one and was redirected to a sales
page. I tried to sign up for the free version and was given 10 credits.
Although that might have covered my needs, I was uncertain what a credit equated to and did not want to incur further risk. Next, I tried Midjourney and was similarly redirected to make an account and shown paid plans. I tried Stable Diffusion because it advertised that it was an open source software so I assumed it would be free, but I couldn’t figure out which software was fit for what I wanted to do. Perhaps my prior experience means I’m able to jump into Adobe products faster, but I found Adobe Express to be the most user and beginner friendly.

Sequence 1: Cupcake Display

Prompt 1: Cupcake display mockup

Prompt 2: Cupcake display mockup for a dinosaur themed birthday party

Prompt 3: Cupcake display mockup for a dinosaur themed birthday party with text banner above that says “happy hatch day”

Prompt 4: Cupcake display mockup for a dinosaur themed birthday party with text banner above that says “happy hatch day” with an orange and red color scheme

Prompt 5: Very large cupcake display mockup for a dinosaur themed birthday party for 50 people with text banner above that says “happy hatch day” with an orange and red color scheme

On first glance, I was really impressed. The number of cupcakes and backgrounds seem impressive, and for the most part, the cupcakes look good in the bottom rows. When I look at the details, there are more issues. The cupcakes in the top two or three rows have more issues with the icing swirl pattern. The banner does not have readable words and the spacing of the flags is inconsistent, but I didn’t expect the AI to do well with words. Finally, the dinosaurs have corrupted details, especially the small dinosaurs on the edges of the table.

I did expect the AI to do better generating the dinosaurs. I wonder if the model intentionally loses detail at the edges to mimic photography lens blurriness. Surprisingly, I think it handled the reflections of the cupcakes on the polished table and on the reflective stand, the texture of the walls, and light and shadows well.

Sequence 2: Cute Capybara

Prompt 1: Cute capybara

Prompt 2: Cute capybara with line drawing for a sticker

Prompt 3: Cute capybara sitting in the bath with an orange on its head; line drawing for a sticker

Prompt 4: Cute capybara sitting in the bath with an orange on its head; line drawing for a sticker; cartoon style

Prompt 5: Cute capybara sitting in the bath with an orange on its head with a baby; line drawing for a sticker; simple vector cartoon style

I am really impressed with this image, but technically speaking, it doesn’t meet all the qualifiers I put in the prompt. The other issue is it doesn’t quite read as a capybara as well as I’d like because the head shape
seems more like a hamster. I’m impressed by the drop shadow it added to make the outline of the sticker stand out. It also did a good job applying a hand drawn style, and there are no weird extra body parts or unnatural shapes.

I wanted something with even less detail and more like a vector with simple, clean lines and little texture. I do think the final image is simplified compared to the prior images, but it still has more detail that I was thinking it would have.

Sequence 3: Convenience Store

Prompt 1: Create an image of a Convenience store

Prompt 2: Create an image of a Japaense convenience store

Prompt 3: Create an image of a Japanese convenience store isometric art

Prompt 4: Create an image of a Japanese convenience store isometric art 2d simplified style

Prompt 5: Create an image of a Japanese convenience store isometric art 2d hand drawn style, blue and yellow color scheme

This prompt progression resulted in many unexpected outputs, but I do like the style and details in the final image. I was not expecting the generator to start with a modern convenience store. In contrast, I was surprised by how traditional the second image was for a Japanese convenience store. I was expecting the model to include Japanese language on the signs but otherwise be similar to the first image.

In the final image, I wanted the generator to have a more 2-dimensional flat style, but it kept a 3d modeled style which I’ve seen in recent advertisements. Although I think the details in this image are more clean and relevant in this image than in others, there are a couple of points where the 3d isometric intersections and depth get confused. Recoloring
seems to be an easy function that the model can do well, but it removes any of the “Japanese” qualities present in previous iterations.

Sequence 4: Dragon

Prompt 1: Drawing of a dragon

Prompt 2: Drawing of a dragon perched on a rock

Prompt 3: Drawing of a dragon perched on a rock in the style of stained glass

Prompt 4: Drawing of a dragon perched on a rock in the style of stained glass in a cathedral

Prompt 5: Drawing of a dragon perched on a rock in the style of stained glass in a cathedral interior mockup

This image does the stained glass style well, however, it was not what I was looking for. I wanted the model to produce a stained glass window image of a dragon but set in a mockup of a photorealistic cathedral. The dragon itself does not seem feasible in stained glass because of the variance in color on the wings which might be because I asked for a drawing. The stained glass does not have an image (e.g. flowers, a cross, a clear geometric pattern, etc.). Because the model didn’t understand what I wanted, the elements have a disjointed, nonsensical composition.

Sequence 5: Flower

Prompt 1: Create an image of a flower

Prompt 2: Create an image of a pop art style flower

Prompt 3: Create an image of a pop art style flower black and white trace; with free space for text

Prompt 4: Create an image of a pop art style flower black and white trace; with free space for text; set in the kitchen

Prompt 5: Create an image of a pop art style flower black and white trace with red accents; with free space for text; set in the kitchen

I was really surprised by this final output because of how it mixed a 2d style with a 3d photo. I wonder if this is because the semicolon separates the qualifiers and if the whole image would be in a 2d traced
style if the prompt didn’t have semicolons. I was surprised that the generator added green details to the trace image and the kitchen was in color. As with previous images, the background details are a little wonky. Upon first glance, things look decent, but when looking more closely, there is an odd table or stool thing to the left, the plants look a bit odd, and there is an ill-defined reflective device at the bottom right which might be a dishwasher but has a strange placement in the kitchen. There is some awkward overlap in a couple of the petals and it’s more obvious
because of the parallel lines defining each petal, but the majority of petals look good.

Reflection

Adobe Firefly is a good tool to create mockups, heavily stylized creative
pieces, or something that only needs to spark a feeling but does not need to be highly accurate to life. Through this process, I’ve noticed that the AI does a good job applying a style and recoloring. It works especially well if the prompt includes a genre of design to reference and mimic, such as a sticker mockup. But the model doesn’t always understand the context of what the prompt is asking for and sometimes ignores qualifiers included in the prompt. The model also struggled to get details that are feasible in life and was totally incapable of creating correct text. Overall, Adobe Firefly gave me some combinations and images that I never would have thought to create, however, it’s difficult to generate an output when I have specific, inflexible vision.

I wouldn’t trust AI to create anything technical. It may be useful to
generate a general base image that is edited or annotated further,
however, given the penchant for losing details, making up details, or
merging details unrealistically. For example, Adobe Firefly could generate a picture of a mountain that passes non-expert inspection, but if that image were for a geography book, the AI image might have a shape that’s not possible to be created by geographic forces. It would be unusable to annotate as an example of geographic processes. I believe AI is more useful to make edits on an existing photo such as generative fill or removing details.