How to Use AI to Generate Textures for Video Games

📌 Textures used: Poly Game Texture Collection

Image 1: 3D video game elements whose textures were rendered with the aid of AI.

Artificial intelligence (AI) has been a staple of video games for nearly as long as there have been games. From the golden age of arcade games in the 1970s and 1980s — think the blocky aliens in Space Invaders and the brightly colored ghosts in Pac-Man — to the 3D pixelated monster dungeons and abandoned facilities of Doom and Quake in the 1990s, and on to the present day, as we watch the emergence of the next generation of real-time rendering and simulation.

Image 2: 3D video game elements with digital textures.

In an era of interactive photorealism, physically based rendering, and virtual immersion, the modern game developer certainly has their work cut out for them. But what if there was a way to use AI not only to engage players during their runtime experience, but also to actually help the creators as they make their games?

That’s where Poly comes in.

Image 3: A digital video game texture.

Far removed from the memory-space limited consoles of yesteryear, modern 2D and 3D game worlds feature increasingly complex visual fidelity with high-resolution textures and state-of-the-art special effects. Even “retro” video games, which recall the creative graphics and nostalgia of previous decades, often make use of highly stylized rendering, mixing the old and new to create something truly unique.

Images 4 and 5: 3D video game elements with digital textures.

Recently, generative AI has taken center stage for its ability to leverage machine learning to produce unique content based on user prompts. This post will demonstrate how you can use this technology to render custom, high-definition visual content — PBR maps like albedo, normal, and depth, roughness, metalness, and even patterns for shader effects — for your game:

Image 6: 3D video game elements with digital textures.

Step 1: Use Poly to Create an AI-Generated Texture

Start with Poly’s Gallery of existing textures — browse for inspiration or search for specific materials by typing in the text box. In this example, we are creating a Minecraft-inspired wood texture.

Image 7: Poly’s Gallery — enter a term in the text box to browse texture types.

If you can’t find what you’re looking for, click on Generate to create a texture from scratch.

Image 8: Poly also lets you generate a texture from scratch.

Once you’re in the Poly editor, you’ll be presented with a full range of tools that allow you to create customized HD textures with 32-bit PBR maps in seconds, using a simple text or image prompt.

Image 9: Download your Poly texture as a .zip file.

Once you’re satisfied with the results, click on the blue Download button and then on Export (.zip). Save the file somewhere handy, after which you can extract the various PBR maps: color (albedo), normal, displacement (height), roughness, ambient occlusion, and metalness.

Image 10: Your downloaded .zip file will contain your PBR maps.

Step 2: Setting Up a Unity Material

Now make sure to have these files somewhere in your project’s Assets folder in Unity. You can then right-click to create a new Material (Create > Material).

Image 11: The files were saved in a Unity project’s Assets folder

Image 12: Right-click to create a new material.

The normal map may not be imported correctly, but you can easily fix this by clicking on the texture and changing the Texture Type setting from Default to Normal Map. Make sure to click on Apply when you’re done. (Unity will remind you to do this when you navigate away from the Inspector.)

Image 13: Set Texture Type to Normal Map if the normal map is not imported correctly.

Render Pipeline

Depending on which render pipeline you’re using, you’ll see a few different options at this point for the default Standard PBR/Lit material Unity will create for you.

1. Built-In (Default / ”Legacy” Pipeline)

Assign the color_map to Albedo, normal_map to Normal Map, and height_map to Height Map.

2. URP (Universal Render Pipeline)

From the Surface Inputs dropdown menu, drag color_map to Base map, normal_map to Normal Map, and height_map to Height Map.

3. HDRP (High-Definition Render Pipeline)

HDRP is similar to URP, but there may be an extra step for using height_map.

To make use of height_map, open the Surface Options dropdown menu and set Displacement Mode to Pixel Displacement.

Under Surface Input, you’ll now see Height Map.

Step 3: Applying the Material to a 3D Model

And that’s it! You can now tweak your material’s Metallic and Smoothness properties, along with anything else, and assign it to a Mesh Renderer of your choosing. We added it to a sphere for previewing our material.

Image 21: We added our material to a new sphere.

Image 22: Our texture applied to two Unity objects.

We hope this guide has given you an idea of just how easy it is to start creating your own PBR library of materials for your next interactive/game project in Unity, Check out Poly!