Altering Images with Masked AI

Oct 11, 2022 at 11:00 am by Warlord720


It wasn’t that long ago, last month about this time, that I wrote about Stable Diffusion and how to run it on your own computer.  The article is mainly about Graphical User Interfaces (GUIs) that give us mere mortals access to the Stable Diffusion AI model without having to jump through technical hoops that would have been anything but clearcut to the casual user. It would also be nearly impossible to one that has no idea what GitHub and Google Colaboratory are.

So here we are at an updated version of the NMKD GUI, one of the simplest, easiest, and most reliable of the 3rd party GUIs I’m aware of. We are now at version 1.5.0 but not that long ago in version 1.4.0 the developer introduced masked-based inpainting. Meaning we can load or drop an image into the GUI and mark an area with a basic black mask (and it is basic) that focuses the generated image into the area of the mask.

Edit with a MaskThe original image. With faces masked out and replaced in the middle image, and bodies masked out in the last image.

Let’s get the technical stuff out of the way. As of this writing, the NMKD Stable Diffusion webpage states the following about the 1.5.0 update:

Features:

  • All dependencies included, no complicated installation
  • Supports text-to-image and image-to-image (image+text prompt)
  • Run multiple prompts at once
  • Built-in image viewer showing information about generated images
  • Built-in upscaling (RealESRGAN) and face restoration (CodeFormer or GFPGAN)
  • Option to create seamless (tileable) images, e.g. for game texture
  • Supports loading custom concepts (Textual Inversion)
  • A variety of UX features
  • As fast as your GPU (2.6 seconds per 512x512 image on RTX 3090)

System Requirements:

  • Windows 10/11 64-bit
  • Recent Nvidia GPU, preferably RTX
  • 16 GB RAM
  • 3 GB VRAM (for slow low-memory mode) or 6+ GB VRAM (for fast mode with more features)
  • ~11 GB disk space after installation (on SSD for best performance)

 Click here for NMKD Stable Diffusion GUI

Now let’s get back to the fun part of all this, the masked inpainting feature that lets us alter images with a combination of masking, the input image, and AI machinations. This is a very simple tool to use for the most part, but it can also be a bit tricky with some prompt tweaking necessary. It is not unusual to make several runs with prompt refinements before you find one that is satisfactory.

For the most part, I used it only in small replacements like face replacement on an existing image however I did try a few crowd and playing field replacements getting decent results for a rookie in the process.

Easy and Quick Face ReplacementQuick and easy face replacement process.

The main point to remember is to not be limited by just one or two runs. You can use an existing image or an AI-generated image that you can then load into the interface, mask, and make several runs until you get what you are looking for.  You can also take that image and change parts of the background or foreground by using it as a reference and masking out the areas you want to change.

At the start, you might want to set your Initial Image Strength high, so the image has more influence over the end results. When I am working with a masked area, I usually set the Initial Image Strength very low, even as low as .50 so my written prompt for the masked area will have more influence in its replacement.

While I am only showing rudimentary examples here you can generate a much more controlled image by making prompt changes and using multiple runs through the AI model with the masks. Sometimes you have to tweak and tweak to get the image you are looking for, or, in some cases, get an image that is close to what you are looking for.

The masked inpainting at this point is very simple and can be a bit frustrating until you dial in the replacement generations but once done, you might be amazed at what you can create without ever touching Photoshop or related tools.

Face ReplacementsOriginal image from Lexica.art, Faces replaced in subsequent images.

 


M.D. McCallum, aka WarLord, is an international award-winning commercial graphics artist, 3D animator, published author, project director, and webmaster with a freelance career that spans over 20 years.  Now retired, M.D. is currently working part-time on writing and select character development projects. You can learn more about MD on his website





This website uses cookies to ensure you get the best experience possible More Info
Got it!