Author: Juju

Posted on

Final Project Documentation

Link to final presentation slides: https://drive.google.com/file/d/1vmSHhVKNERSHDP3kohUbkKPy6XJEMXsV/view?usp=sharing

Link to final project posters: https://drive.google.com/file/d/15ZaGm7wcuEwJYs4QItOdXVyZF-wcR_OH/view

Link to final project video: https://www.youtube.com/watch?v=dNwHWeazyxk

Link to concept presentation slides: https://drive.google.com/file/d/1jgxeo-knGx7nLrnWBmPZYLOIwz4mdJ8k/view?usp=sharing

Background

For my final project, I’ll be exploring abstract art with Deepdream in two different mediums, video and print. I plan on creating a series of images (printed HD onto posters to allow viewers to focus on the detail) and a zooming Deepdream video. I’ll create the original images with digital tools, creating abstract designs, and then push them one step (or several) further into the abstract with the Deepdream algorithm.

I love creating digital art with tools such as Photoshop, Illustrator, Data moshing and code such as P5, however I’ve realized that there are limitations to these tools in terms of how abstract and deconstructed an image can get. I’ve also been interested in Deepdream styles for a while and the artistic possibilities, and I love the infinite ways Deepdream can transform images and video. In the first week, I presented Deepdream as my case study, using different styles of the Deep Dream generator tool to transform my photos.

Case study slides: https://drive.google.com/file/d/1hXeGpJuCXjlElFr1kn5yZVW63Qcd8V5x/view

I would love to take this exploration to the next level by combining my interest in abstract, digital art with the tools we’ve learned in this course.

Motivation

I’m very interested in playing with the amount of control digital tools give me to create, with Photoshop and Illustrator giving me the most control over the output of the image, coding allowing me to randomize certain aspects and generate new designs on its own, and data moshing simply taking in certain controls to “destroy” files on its own, generating glitchy images.

However, Deepdream takes away all most of this prediction and control, and while you are able to set certain guidelines, such as “layer” or style, octaves, or the number of times Deepdream goes over the image, iterations and strength, it is impossible to predict what the algorithm will “see” and produce in the image, creating completely unexpected results that would be nearly impossible to achieve in digital editing tools.

References

I’m very inspired by this Deepdream exploration by artist Memo Akten, capturing the eeriness and infinity of Deepdream: https://vimeo.com/132462576

His article (https://medium.com/@memoakten/deepdream-is-blowing-my-mind-6a2c8669c698) explains in depth his fascination with Deepdream, something I share. As Akten writes, while the psychedelic aesthetic itself is mesmerizing, “the poetry behind the scenes is blowing my mind.” Akten details the process of Deepdream, how the neural networks work to recognizevarious aspects of the reference image based on its previous training and confirm them by choosing a group of neurons and modifying “the input image such that it amplifies the activity in that neuron group” allowing it to “see” more of what it recognizes. Akten’s own interest comes from how we perceive these images, as we recognize more to these Deepdream images, seeing dogs, birds, swirls, etc., meaning viewers are doing the same thing as the neural network by reading deeper into these images. This allows us to work with the neural networks to recognize and confirm the images it has found, creating a cycle requiring both AI technology and human interference, to set the guidelines and direct it to amplify the neural network activity as well as perceive these modified images. Essentially, Deepdream is a collaboration between AI and humans interfering with the system to see deeper into images and produce something together.

Experiments

I started by playing around with different images I had already created in code and with Illustrator and Photoshop using Alex Mordvintsev’s Deepdream code to find the settings I wanted to use. While at first I went all out with the strength and iteration settings, I soon realized that the images became too obscured and I wanted the final result to be an abstraction of an existing image, not a completely indiscernible abstraction.

Here’s an image created with the 5a layer on high strength with many iterations:

After researching what each setting related to and refining my process, I settled on my favorite settings in Deepdream, such as these settings that I used for the video:

Then I created my own images for the posters and video, using what I learned from my experiments. I realized that images with a lot of white space often had less interesting results and use of color, such as this image of circles created with P5:

And these results, which didn’t achieve the style I was going for:

Therefore, I focused on creating images with a full canvas of color and/or detail to get the most interesting abstract results. I settled on these 5 images to create the Deepdream posters.

This image of “Juju’s” created in p5 which presented an interesting pattern and enough detail across the canvas for Deepdream abstraction:

Here is after 5a level iteration:

After editing the image to turn it black and white for the final result:

I really liked this result because the black and white filter applied in Photoshop combined with Deepdream added texture and depth, even though the original image was 2D. 

The next image I created was this one, beginning from a photograph and using Illustrator. Using the trace image and merge functions, I separated details of one photograph into the top and bottom of the canvas, applying shapes and letters to add detail.

Using the layer 3b I achieved this result:

I like this one because the abstraction still preserves the “dual reality” I created in the first image in Illustrator, adding fascinating detail and depth.

For the next one, I started with this photo of my old laptop’s lid with stickers on it:

I then used Photoshop to distort the image:

In Illustrator I traced, expanded, ungrouped and moved around various parts to abstract it further:

Then, using the 3b layer in Deepdream, I created this image, with some connection to reality but only if the previous progressions are viewed together:

For this image, I took a photo of a pattern on my pants and used Photoshop to edit the colors and distort the “pattern”:

I then used layer 3a to create this stunning image with new colors and seemingly infinite detail:

Finally, I created this image in p5 using beginShape and Perlin noise to randomize the drawing of the shapes and the colors:

Using layer 3a with a lower strength to highlight the rectangle lines in the original, I created this final image:

A link to all my final images is at the top of this post. To create the posters in A3, I formatted the images in Illustrator, adding quotes to highlight my view of my relationship with Deepdream and creation throughout the process. While I originally viewed this project as giving up control to Deepdream to abstract images further, I ended up regaining that control through editing photos and adjusting Deepdream settings to achieve the level of abstraction I wanted. This made me view the process more as a collaboration, using Deepdream for my specific artistic purposes instead of giving it control over my images.

Social Impact

I think exploring Deepdream is a great case study in the societal impact of AI technology, especially when thinking about Deepdream’s original uses and its artistic applications. Developed originally to enhance Google image’s perception of the content of images, developers realized the potential to create their own psychedelic imagery by increasing the strength, iterations and octaves and pulling images from different layers of neural network to reveal the algorithm’s process of enhancing and “seeing deeper” inside of given images. As AI is developed for practical uses of enhancing companies’ performances and governments’ efficiency, I foresee artists continuing to explore the artistic applications of various AI developments by tweaking the software for their own uses. This shows that as capitalism monetizes technological functions, humans will continue to see the beauty and aesthetics of these functions. Deepdream alone has much more potential in artistic applications as artists continue to explore its uses.

Further Development

One way I plan to develop this project further is through VJing with Deepdream videos. After creating my zooming Deepdream video set to music, I realized that it had great potential to be used as a video DJing tool accompanying techno music. I also received this feedback from friends I showed the video to who thought the pairing with music added a new element to the music. This winter I plan on throwing some parties with my friends who are DJs, and I’m planning on exploring how to import Deepdream style videos into VJing software to allow manipulation of the videos in real time with music. I liked using video editing techniques and manipulating the speed, colors (such as inverting the colors in time with the rhythm) and zoom to accompany the music and I see a lot more potential in this area.

I also plan on continuing my Deepdream poster series as a personal project since I love the results and I have many more images I’ve created with code and digital editing tools I would love to explore more. I plan on creating a new page on my portfolio website showcasing my AI art and potentially collaborating with other artists to use Deepdream in further projects.  

Posted on

Week 12: Final Concept Documentation

Link to concept presentation slides: https://drive.google.com/file/d/1jgxeo-knGx7nLrnWBmPZYLOIwz4mdJ8k/view?usp=sharing

Background

For my final project, I’ll be exploring abstract art with Deepdream in two different mediums, video and print. I plan on creating a series of images (printed HD onto posters to allow viewers to focus on the detail) and a zooming Deepdream video. I’ll create the original images with digital tools, creating abstract designs, and then push them one step (or several) further into the abstract with the Deepdream algorithm.

I love creating digital art with tools such as Photoshop, Illustrator, Data moshing and code such as P5, however I’ve realized that there are limitations to these tools in terms of how abstract and deconstructed an image can get. I’ve also been interested in Deepdream styles for a while and the artistic possibilities, and I love the infinite ways Deepdream can transform images and video. In the first week, I presented Deepdream as my case study, using different styles of the Deep Dream generator tool to transform my photos.

Case study slides: https://drive.google.com/file/d/1hXeGpJuCXjlElFr1kn5yZVW63Qcd8V5x/view

I would love to take this exploration to the next level by combining my interest in abstract, digital art with the tools we’ve learned in this course.

Motivation

I’m very interested in playing with the amount of control digital tools give me to create, with Photoshop and Illustrator giving me the most control over the output of the image, coding allowing me to randomize certain aspects and generate new designs on its own, and data moshing simply taking in certain controls to “destroy” files on its own, generating glitchy images.

Created with p5 with randomization techniques and Perlin noise:

Created with datamoshing, filters and layout in P5:

Abstract image created in Photoshop:

However, Deepdream takes away all most of this prediction and control, and while you are able to set certain guidelines, such as “layer” or style, octaves, or the number of times Deepdream goes over the image, iterations and strength, it is impossible to predict what the algorithm will “see” and produce in the image, creating completely unexpected results that would be nearly impossible to achieve in digital editing tools.

References

I’m very inspired by this Deepdream exploration by artist Memo Akten, capturing the eeriness and infinity of Deepdream: https://vimeo.com/132462576

His article (https://medium.com/@memoakten/deepdream-is-blowing-my-mind-6a2c8669c698) explains in depth his fascination with Deepdream, something I share. As Akten writes, while the psychedelic aesthetic itself is mesmerizing, “the poetry behind the scenes is blowing my mind.” Akten details the process of Deepdream, how the neural networks work to recognizevarious aspects of the reference image based on its previous training and confirm them by choosing a group of neurons and modifying “the input image such that it amplifies the activity in that neuron group” allowing it to “see” more of what it recognizes. Akten’s own interest comes from how we perceive these images, as we recognize more to these Deepdream images, seeing dogs, birds, swirls, etc., meaning viewers are doing the same thing as the neural network by reading deeper into these images. This allows us to work with the neural networks to recognize and confirm the images it has found, creating a cycle requiring both AI technology and human interference, to set the guidelines and direct it to amplify the neural network activity as well as perceive these modified images. Essentially, Deepdream is a collaboration between AI and humans interfering with the system to see deeper into images and produce something together.

Posted on

Week 11: Deepdream exploration

For my deepdream exploration, I created a zoom video that’s a compilation of two deepdream sessions. I created one deepdream with setting mixed4c, then another with the last image picked up from the first video and zoomed in further, also on mixed4c.

deepdream zoom low quality

I had a lot of challenges with this, as at first I didn’t realize I need to change the video file name to save the video (leading to some confusion) and I also had some random errors where img0.jpg wasn’t defined as I didn’t realize that I had to run all the cells together every time for everything to work. I also had trouble with the original photos I tried to use as they were very high quality and I didn’t realize that the session would close down after 30 minutes so the videos kept failing to generate and getting stuck loading at one or more of the cells.

After getting some help from Aven I was able to realize these errors and generate the videos. I plan on using this technique for my final video project, so I’m excited to explore it more!

I also put together the class video for the bigGAN collaboration, available here:

https://www.youtube.com/watch?v=xxrblvC-VOM 

Posted on

Train and Inference Style Transfer

I had a lot of trouble with this assignment and ran into a few difficulties. In the end, I wasn’t able to train the Style Transfer despite overcoming most of the challenges I faced. However, I do feel like I learned much more about the files and the process in terminal after spending hours problem solving and researching the challenges I faced.

To start training, I cloned the repo, created the environment and checked the installation, but had to use a different command from the slides because I realized that the folder was named styleTransfer instead of styletransferml5, something which threw me off on several steps of the training. After successfully downloading the setup, I renamed my image, shapes.png, an interesting 3D shapes image I thought would produce interesting results, and loaded it into the images folder on my devCloud.

After completing these steps, I attempted to run the qsub -l walltime=24:00:00 -k oe train.sh code, but didn’t receive the “successfully training” message. Instead, I discovered that every time I ran the code, the program created a new “training” file, such as train.sh.o411228, and an error file, train.sh.e411228.

Initially I didn’t realize that train.sh.e411228 was an error file, so I kept checking the other version to find confusing outputs such as this:

Checking the error file showed this error, saying that style.py didn’t exist, so I reuploaded the style.py into the train_style_transfer_devCloud folder on devCloud and kept trying again.

At this point I reviewed all my steps on the slides and realized that I needed to edit the train.sh file, as I had not realized that before. I went back into the downloaded folder from Github and changed the username and file name, but after a few more error messages I realized that the original train.sh indicated the folder was styletransferml5, which is why the program couldn’t find the style.py file, as it was under activate styleTransfer in the train_style_transfer_devCloud folder. Here is my edited train.sh file:

After correcting my mistake, I reuploaded the train.sh file again and submitted the qsub -l walltime=24:00:00 -k oe train.sh command, and was very excited to see the message, ml5.js Style Transfer training!

However, after a day and a half went by, the message was still the same with no successful models trained. At this point you directed me to check the error file, which showed this new error, train path not found.

The final hurdle was realizing that when cloning the repo the unzip train2014.zip had failed to work, which meant that despite problem solving earlier the file was still zipped and therefore couldn’t be used to train. I redownloaded the file again and attempted the use the unzip command in Colfax, but got these error messages after several attempts.

In the end, I kept getting the “broken pipe” message, so I had to give up my attempts. While I feel like I got very close and got to know all the parts that go into style transfer training, experiencing all the complications was very frustrating especially after spending hours problem solving, getting help from friends and fellows, and feeling so close to the finish line after the success messages.

Posted on

Midterm Writing Assignment

In my Week 07 documentation, I rewrote background and inspiration to clarify, so I’ve posted my Week 07 version of these along with the social impact, artistic influence and future plans.

Is it possible to teach an AI design? This is a problem at the forefront of Machine Learning exploration and a fascinating topic that could change how designers work and how companies operate. For my midterm project, I wanted to examine ways to create an algorithm that could design posters based on a specific style and user inputs. In the end, I believe that my proposed idea is  likely possible, but would require time, extensive data and skills beyond my range.

Background

I wanted to do this project because of my interest in design, as well as the Swiss Minimalist style, which I chose to focus on. Swiss minimalism, particularly the poster style I chose to focus on, was developed in Switzerland during WWII and made famous by designer and author Josef Muller-Brockmann, editor of magazine Neue Grafik. Brockmann’s posters used bold colors, a grid layout and simple, clean typography to create aesthetic posters. Brockmann’s style changed graphic design, ushering in an appreciation for layout and typography. His style is time-tested and still favored today by artists and institutions.

Another source of inspiration is the class Programming Design Systems, which I’m taking this semester with Cici Liu. Before this class, I mostly used Adobe Illustrator, Photoshop and InDesign to create designs, which often involves a lot of manual work, i.e. physically moving things around the canvas and placing them myself to see what looks good. I love making posters, whether it’s for a party I’m throwing, part of my freelance work or just for fun, but I often find myself saving countless versions of the same poster with slightly different positioning and elements. Through the class, particularly the poster-designing assignment, I found it fascinating to see how code could be used to simplify the design process, and how an aesthetic poster could be broken down into coding data. We work in P5, which can also be used to randomize posters by setting limits and creating arrays with sets of data to allow the code itself to generate many different versions of posters within set ranges.

I used P5 to recreate a few Brockmann posters and realized the Swiss Minimalist style poster could be entirely recreated with P5, as the posters usually involve a few headings of text (sometimes rotated or overlapping), use of simple shapes, curves and lines and a grid layout, all elements that can be translated into code to reproduce the posters.

Inspiration

As mentioned in my project proposal, I was inspired by Alibaba’s LuBan AI software, which is used to generate banners and ads for products on Taobao and Tmall. Most of the pop-up ads in the app are now generated by the software, which can generate 8,000 banners per second. This AI was trained with a more complex process involving teaching it design elements separately, and then having teams of designers use reinforcement training to steer the AI towards “good” designs. The AI was trained on copy, product shot, background, logo and decorating artifacts, all the elements needed in a simple ad, from a huge dataset of these individual features. This project took years and likely thousands of programmers and designers, but it shows that a Machine Learning algorithm can be taught design elements and produce designs indistinguishable with those from a designer.

Process

After bringing Aven my idea of teaching a Machine Learning algorithm poster design, we brainstormed possible ways of doing the project. Inspired by the Programming Design Systems class, I was interested in using P5 as the way the algorithm could “design” the posters as that way the posters would be represented as data. This means that I could use P5 “data” to teach the program, and it could create an algorithm based on the data to design original posters with Swiss Minimalist design principles. My goal by then for the midterm was to find an effective way of creating and gathering this data.

Aven said I would need at least 1,000 posters, so I started my research, saving posters that could be turned into data (eliminating those with images and design elements that couldn’t easily be recreated with code) and saved several hundred minimalist posters.

I began to notice that many of the posters had 3 main types of text, which I began to call Heading1, Heading2 and Heading3. Heading1 was usually bold and almost decorative, often very large and distinctive on the poster. Heading2 and 3 usually had smaller font and creative placement, containing the actual information for the poster. The posters often used one or two types of shapes, repeated and in various colors. After observing these posters, I created an Excel spreadsheet for the data I wanted to gather and broke it down into a few different categories that I thought could represent most of the posters:

Heading1 text placement (location on the canvas, text size, text color, text rotation, and the same data points for Heading2 and Heading3, then Shape1 placement, size, fill, rotation and stroke, continued for the rest of the shapes present on the poster, and background fill. I was planning to input the data as P5 code.

I then brainstormed the user interface, which I wanted to be simple but allow users to create their own personalized posters. While my original idea involved different “styles,” I decided to simplify that to “black and white” or “color.” I wanted to allow users to input the three headings themselves, perhaps with a guideline indicating that Heading1 should involve little text and Heading3 should contain the main data for the poster.

Potential solutions and challenges

The first major challenge I encountered was gathering the data. While I knew what data I wanted to pull from the posters and I had the posters to use for this, I quickly realized that placement, rotation and fill weren’t that intuitive. In order to get this data, I would have to “redesign” each poster in P5. Some of the posters didn’t share the same dimensions, so I would have to reformat the posters to one dimension to create consistent data. I would also have to use color pickers online to find the colors inside the posters, and then pick a consistent color system, such as HSB in P5, to represent the colors. Rotation also presented a challenge as it would take time to find the rotation and placement, which I would have to do by essentially “eyeballing” the poster and trying to match the coded version to the real one. Also, recreating curved lines with Bezier curves would prove a major challenge, as well as any irregular shapes that would involve more coded data than I could easily input. I quickly realized that turning 1,000 posters into P5 data was a huge task that would take many hours and could produce imperfect results.

The second major challenge was how to input this data into the Machine Learning algorithm. I met with Aven again to discuss this challenge, and he suggested JSON, as I would be able to upload my Excel spreadsheet of data into the platform and then import JSON into the model to train it.

Also, since I wouldn’t be using a simple pre-trained model, I would need to find a program that could read and learn Swiss Minimalist principles from the data, and then be able to generate new P5 data and create a sketch for a new poster. This seemed very challenging as we discussed how we didn’t know whether such a program already existed, so I may get through all the steps of data gathering to realize that I wouldn’t be able to train it at all.

In order to create the user interface, I would also have to find a way to input the user’s headings into the P5 sketch while allowing the algorithm to generate the rest of the design based on the data, and then display that sketch. This also posed a challenge as while I have experience in P5, linking all the systems together to create the project I wanted would involve many steps I wasn’t familiar with.

After reaching the conclusion that turning thousands of posters into data would be beyond my ability this semester, I started looking into other ways of finishing the project, such as human in the loop. I knew from my experience with P5 that I could use several different sketches to produce infinite posters in the Swiss Minimalist style by randomizing certain elements within ranges. With this in mind, I was curious whether I could have feed an algorithm these sketches and have it produce its own random designs, which I could then evaluate in order to essentially teach it to produce more of the “good” designs. This is very similar to LuBan’s reinforcement process, where teams of designers evaluated the AI’s designs in order to teach it the finer points of design. While Swiss Minimalism famously uses grid systems, which generally follow “rules,” many of the posters also “break” these grid system rules in order to create interesting and unexpected designs. This would be one aspect I could teach the algorithm through human in the loop: when to “break” these rules and when to follow them in a design to create aesthetic results. One challenge however is that most of the resources on human in the loop that I came across start with a high quality training dataset, and I couldn’t find references on whether I could have the algorithm randomize posters from different ranges of data and elements to create its own dataset. I briefly considered somehow gathering the data from different iterations of P5 sketches to create a database, but if all the sketches use the same code to create different posters, this would also prove to be a huge challenge.

With this is mind, I conclude that while this project may be possible if a team can turn thousands of Swiss Minimalist posters into P5 data and then teach an algorithm to use that data to generate its own posters in P5, it would take time, resources and knowledge out of my reach in order to do so. Also, at any step of the way if something went wrong or if the actual algorithm needed doesn’t exist, this could easily fail to produce the results I was looking for. This project was a fascinating exploration for me into the world of building databases and training models, and while in the end I’m not able to create the project I wanted to, I have a much better grasp on the realistic challenges of data, training and algorithms, which I can take with me for my next, more achievable, project.

Social impact

I think this project could have a large social impact as AI is currently changing how designers work in many companies, and the future of design almost certainly includes collaborating with AI to streamline the design process. In terms of poster design specifically, I could see online design services like Canva incorporating AI programs that allow users with limited design experiences to simply input their poster’s headers and style and immediately generate a well-designed, unique poster. This would change the accessibility of well-designed materials, as well as the speed, since anyone would be able to enter text and generate simple posters immediately.

Artistic influence

If an AI can learn layout and basic design principles, this could have many artistic uses as many pieces of visual art rely on certain visual principles such as the golden ratio and placement. Using the program I want to create, the algorithm could be trained to produce abstract visual art by learning P5 sketches with abstract shapes, patterns and colors and producing its own. It would be fascinating to see how AI produces art in different abstract styles and how different styles can be boiled down into algorithms. For example, many of Wassily Kandinsky’s abstract paintings can be broken down into shapes in P5, making it possible to turn them into data and train an algorithm on the paintings themselves. While the hurdle may be that there are not enough paintings by one particular artist to create a reliable dataset, perhaps a dataset could be created on a particular style of painting to train the AI.

Future plans

In the future I would love to expand the styles from Swiss Minimalism into Bauhaus, Vaporwave and more artistic styles with sets of design principles. In order to do this, I would also want to find a different way of turning posters into data besides P5 code, as it would allow for a lot more flexibility for visual elements. An interesting addition would be allowing photos and training the algorithm on photo placement and scale, so that users could upload photos into the interface and have them placed on the poster by the AI.

Sources

https://www.figure-eight.com/resources/human-in-the-loop/

Swiss (International) Style Of Design: The Guiding Principles That Influence Flat Design

https://medium.com/@rexrothX/ai-visual-design-is-already-here-and-it-wont-hesitate-to-take-over-your-petty-design-job-934d756db82e

https://medium.com/@deepsystems/human-in-the-loop-for-object-detection-with-supervisely-and-yolo-v3-fa205ff07c1f