A Controllable Appearance Representation for
Flexible Transfer and Editing

Material appearance is shaped by complex interactions between several factors, including surface properties, lighting, geometry, or viewing conditions [fleming03JoV, lagunas2021joint], often requiring high-dimensional data for accurate modeling. A number of works have attempted to find a compact representation of appearance, searching for low-dimensional BRDF embeddings [Matusik:2003], enabling applications such as compression or editing. However, they usually require costly human-annotated datasets [serrano2016intuitive, Toscani2020ThreeDim, Shi2021LowDim], or produce spaces that lack interpretability because their focus is on some other aspect, such as compression [Hu2020DeepBRDF, Soler2018Manifold, Zheng2022NeuralProcesses]. While the former work in BRDF space, a series of approaches have focused on working directly in image space, to account for the interplay of confounding factors like geometry or illumination in the final perception of appearance [lagunas2019similarity, Serrano2021_Materials]. Still, they rely on supervised learning-based methods and require large amounts of human annotations, which can be partially alleviated with weak supervision [guerrero2024predicting]. Some methods have focused directly on material editing in image space [Delanoy2022generative, subias23in-the-wild_editing], leveraging GAN-based frameworks to allow controlled modification of specific attributes, like glossiness or metallicness, and also requiring ground-truth human labels for training. Unsupervised learning of appearance representations has been mostly limited to specific attributes, like gloss [storrs2021unsupervised] or translucency [liao2023unsupervised], training on datasets with limited variability. In this work, we present a self-supervised model that learns a highly-compact latent space of material appearance. Remarkably, our model can disentangle interpretable factors like color or glossiness in images depicting homogeneous real-world materials, without any prior knowledge or annotated data.

Disentangled representation learning [wang2024disentangled, locatello2019challenging] aims to separate underlying factors of variation within data, improving interpretability in generative models. Variational Autoencoders (VAEs), including $\beta$VAE [higgins2017beta], achieve this by balancing reconstruction fidelity and latent space regularization, while FactorVAE [kim2018disentangling] introduces a Total Correlation (TC) penalty via a discriminator network to enhance factor independence. Extensions like $\beta$TCVAE [chen2018isolating] propose different implementations of this TC term. A key challenge in these models is the posterior collapse, where latent variables lose informativeness by matching the prior too closely. Due to its relevance, this issue has been widely addressed [sonderby2016ladder, fu2019cyclical, yan2020re, kinoshita2023controlling]. Beyond VAEs, GAN-based [chen2016infogan] and diffusion models [yang2023disdiff] have also been explored for disentanglement. Nevertheless, the explicit modeling of a latent space in VAE-based models facilitates learning independent factors within a compact representation. Closer to our work is that of Benamira et al. [benamira2022interpretable], that used $\beta$VAE to disentangle material appearance from measured BRDFs. In contrast, our model disentangles material appearance in image space, accounting for the influence of factors like illumination or geometry. We adapt FactorVAE to mitigate posterior collapse, and enable a self-supervised disentangled latent space to encode and modify material appearance (see Sec. 3).

Since the seminal work from Ho et al. [ho2020denoising], diffusion models have revolutionized image generation with exceptional quality and diversity, by progressively denoising from random noise [saharia2022photorealistic, betker2023improving, rombach2022high]. Conditioning mechanisms have recently played a central role to expand the functionality of diffusion models for controlled generation and editing. These include techniques like ControlNet [zhang2023adding] for multi-modal conditioning, LoRA [hu2021lora] for task-specific fine-tuning, or IP-Adapter [ye2023ip] and T2I-Adapter [mou2023t2i] for image-based conditioning, by injecting features from reference images alongside text prompts to influence the generation.

In the context of material appearance, diffusion models have been used for material generation [zhang2024dreammat], physically-based synthesis [vainer2024collaborative], material capture [vecchio2024controlmat], or texture editing [guerrero2024texsliders]. In ColorPeel [butt2024colorpeel], they condition diffusion models on disentangled properties like color and texture, allowing for fine-grained editing but requiring labeled data to train the conditioning, which complicates generalization to novel properties. Cheng et al. [cheng2024zest] demonstrated zero-shot material transfer by injecting CLIP [radford2021clip] embeddings of reference materials into a diffusion pipeline, achieving compelling results without any additional model training. However, their reliance on CLIP limits disentanglement between material properties and additional factors, such as lighting and geometry. Alchemist [sharma2024alchemist] showed impressive performance on material editing in image space, training diffusion models for specific material attributes with supervised ground-truth labels.

In our work, we train an IP-Adapter to condition a pre-trained diffusion model based on Stable Diffusion-XL [podell2023sdxl] on our compact self-supervised appearance representation, to showcase proof-of-concept applications of it (namely, appearance transfer and editing).

We employ an encoder-decoder architecture to create a bottleneck of reduced dimensionality that encapsulates the internal representation of appearance (Fig. 1). We build our model on FactorVAE [kim2018disentangling], a variational autoencoder for self-supervised learning of disentangled representations, for its effective balance between disentanglement and reconstruction quality. We introduce two key modifications to the original FactorVAE to make it suitable for our goal: (1) we modify the loss to avoid posterior collapse (Sec. 3.1.1), and (2) we input geometry information to the model in the form of normal maps, compelling the latent space to focus on appearance (Sec. 3.1.2).

Our model is trained in a self-supervised manner, learning to apply an input appearance onto a reference geometry, while building a well-structured latent space by optimizing the loss function (Eq. 1).

Existing datasets of material appearance are relatively abundant [storrs2021unsupervised, Delanoy2022generative, Serrano2021_Materials]. However, they often include simple and unrealistic geometries with limited diversity, or are highly unbalanced with respect to appearance factors such as glossiness or hue. This is particularly problematic for our self-supervised training, as it could introduce unintended biases into the learned appearance representations. Moreover, we seek for a larger-scale dataset to improve generalization.

Therefore, we carefully designed a training dataset comprised of 98,550 synthetic images of 30 different objects rendered with 365 measured BRDFs [Matusik:2003, Dupuy2018Adaptive, Serrano2021_Materials]. In order to facilitate the understanding of illumination by the network, we systematically vary the lighting for each object and material combination, leading to 9 lighting conditions (9 $\times$ 30 $\times$ 365 = 98,550). A representative set of images of the dataset, rendered with Mitsuba [Mitsuba], can be found in the supplementary material (S2).

Quantitatively, we compare our method to three baselines that are commonly used for disentangled representation learning (Sec. 2.2), all trained on our dataset (Sec. 3.2): $\beta$VAE [higgins2017beta], $\beta$TCVAE [chen2018isolating] and the vanilla FactorVAE [kim2018disentangling]. The comparison evaluates disentanglement, interpretability and reconstruction quality. Our test set is comprised of images from the Serrano dataset [Serrano2021_Materials]. From it, we select the ones featuring simple geometries (blob and sphere) to evaluate disentanglement and interpretability, to ensure that the lack of specialization of the baseline models in terms of geometric disentanglement is not overly penalized. To evaluate reconstruction quality we select images from a complex, unseen geometry (statuette). Note that, while some materials are both in our training and test sets, they are rendered with different illuminations and scene configurations, resulting in different appearance; besides, all baselines we compare to are trained and tested on the same sets.

Disentanglement is typically measured quantitatively with Gaussian Total Correlation (GTC) [Watanabe1960information] and the Mutual Information Score (MIS) [shannon1948mathematical], which analyze the structure of the space and do not require labeled data. Measuring interpretability does require labeled data, and we measure it with the Z-min [kim2018disentangling] and Mutual Information Ratio (MIR) [whittington2023disentanglement] metrics, using ground truth labels available in the test dataset. We show all four metrics in Table 1. Our model clearly surpasses the rest, including the vanilla FactorVAE, showing the benefits of our modifications described in Sec. 3.1. Reconstruction quality is evaluated on the same test data with three widely-used metrics: SSIM [Wang2004SSIM], LPIPS [zhang2018perceptual], and PSNR. Our model achieves better reconstructions, which can be attributed to the inclusion of geometry information in the decoder pipeline.

Qualitatively, we can evaluate the disentanglement and interpretability of our latent space by visualizing the reconstructions generated by our decoder when sampling the latent space. To visualize the information encoded in the space, we take samples along each dimension (keeping the rest at a constant value of zero), generating feature vectors $f\in\mathbb{R}^{6}$ that are fed into the decoder together with a normal map. The results are the images shown in the prior traversals plot in Fig. 2 (the normal map is that of the Havran scene in this case [Havran2016Shape]). In the figure, each row corresponds to the traversal of one latent dimension. We can see how the model identifies a slightly glossy gray material as the neutral one (zero-valued feature vector, central column). We can also clearly observe how, despite the lack of supervision, our model identifies interpretable factors that emerge from the data: Dimension 1 encodes lightness, dimensions 2 and 3 correspond to hue, dimensions 4 and 5 encode lighting direction (top to bottom and left to right, respectively), and dimension 6 represents gloss. Additional visualizations can be found in the supplemental material (S3).

Finally, we evaluate the ability of our model to encode the material appearance of a given image, and to modify such appearance along the dimensions of our latent space. Fig. 3, left, shows, for six real-world, photographed objects (the background has been masked), the result of encoding them with our model and decoding them using the normal map shown. The very similar appearance between the original and the reconstructed result shows the ability of our space to successfully encode the appearance of real objects. In Fig. 3, right, we also encode an input image of a real object into the latent space, obtaining its feature vector, and we then sample this space in each dimension, akin to what was done in the prior traversals plot in Fig. 2, but with the posterior, leading to posterior traversals plots. We can see how the material appearance of the input image is correctly captured (black boxes), and how we can modify the original appearance in a controlled manner by traversing the dimensions of the latent space. However, despite the great performance shown by the FactorVAE encoder in successfully distilling a disentangled and interpretable material appearance representation from the input image, the decoder pipeline proves insufficient when generating images of geometries very different from those seen during training (see supplementary material (S3.3) for more details). This highlights the need of a more powerful framework that translates the feature vector obtained into a final image, which is explored further in Section 5.

Our 6-dimensional space is highly compact, as compared to other alternatives in the literature (e.g., CLIP embeddings [radford2021clip] can be 512D (ViT-B/32, ViT-B/16), 768D (ViT-L/14) or 1024D (ViT-H/14), and StyleGAN-based autoencoders [karras2019style] are 512D and higher), facilitating interpretability. We explore the effect of modifying the dimensionality of our space, aiming to keep a balance between interpretability and disentanglement: larger latent spaces tend to dilute information in more dimensions, penalizing interpretability, while smaller ones need to embed the same information in a more constrained latent space, which often leads to worse disentanglement. We analyze models from three to ten dimensions in the latent space, including quantitative results in Figure 4. Our model with six dimensions achieves the best balance between interpretability and disentanglement, as represented by MIR (higher is better) and MIS (lower is better) metrics, respectively. We also include qualitative results of this analysis in the supplementary material (S3.4).

We evaluate the effectiveness of our design decisions by running ablation studies. We include here ablations on our proposed modifications of the loss function to tackle posterior collapse (Sec. 3.1.1), and on the use of normal maps to enforce the encoding of appearance (Sec. 3.1.2). For additional ablations on the reconstruction loss and normal map resampling, please refer to the supplementary material (S4).

We design a diffusion-based pipeline that uses two sources of information as input: a geometry reference image, which defines the target geometry of the object, and an appearance feature vector in our latent space, which specifies the desired appearance to be applied to the target geometry. An overview of our pipeline is shown in Fig. 5. We use pre-trained latent diffusion model, RealisticVisionXL4.0, which is a fine-tune of the base model Stable Diffusion XL designed for photo-realism. We add our sources of information to condition the generative process through two distinct branches: an appearance conditioning branch and a geometry conditioning one.

The appearance conditioning branch encodes the material and illumination information from one or more reference images into an appearance feature vector by using our encoder (Sec. 3), effectively performing appearance transfer. Alternatively, one can directly sample the latent space to generate such vector. Further, the user can navigate the space, enabling controlled fine-grained adjustments to the generated images (appearance editing). Our appearance encoder, trained as explained in Sec. 3, is kept frozen during the training of the diffusion pipeline. We integrate the information of the appearance feature vector by training an IP-Adapter [ye2023ip]. Following the IP-Adapter implementation, we plug an external network via a cross-attention mechanism, and train it by minimizing the original Stable Diffusion loss [rombach2022high]. Only the weights of the external network are updated during training, in order to preserve the generative capabilities of the base model.

The geometry conditioning branch takes an image as input and incorporates the target geometry information into the diffusion pipeline via a combination of pre-trained ControlNets [zhang2023adding], designed to process Canny edges and depth maps. Depth information helps in transferring the general structure of the shape, while the Canny edges map allows to preserve the high-frequency details. For inference, we automatically obtain the depth map from the input image using a single-view depth predictor [ke2023marigold].

Finally, we follow previous work [cheng2024zest] and use an inpainting base model during inference, which restricts the generation to the area of the object, keeping the background intact. Please refer to the supplementary material for full implementation details and ablations of our pipeline (S1.2).

Our pipeline is therefore versatile, and can be used for different applications depending on how the inputs are configured. Fig. 5 illustrates a canonical use case in which the appearance of the bunny is encoded with our model and transferred to the image of Michelangelo’s David with our diffusion pipeline, performing appearance transfer. Once encoded, given the disentanglement and high controllability of our latent space, the user can easily modify the appearance vector and re-generate the image, thus performing fine-grained appearance editing, as shown. Our pipeline is however flexible to other use cases, such as introducing the same image for both appearance and geometry conditioning to perform direct editing (Fig. 9), selective transfer by extracting different factors of appearance from different input images (Fig. 6 and Fig. A Controllable Appearance Representation for Flexible Transfer and Editing, right), manually defining the appearance of an object by sampling our latent space, or by interpolating in this space (Fig. 7).

We evaluate our appearance-aware diffusion pipeline, showcasing applications of our disentangled appearance representation in different use cases. Additional results of our diffusion-based pipeline can be found in the supplementary material (S6).

The network implementation builds on the work by Dubois et al. [dubois2019dvae], which contains the definition of different VAE-based models. The system is developed using the Pytorch library [NEURIPS2019_9015], and trained on a NVIDIA GeForce RTX 3090. We use the Adam optimizer to optimize both the autoencoder and the discriminator in charge of computing the TC term, with learning rates of $1e^{-3}$ and $7e^{-5}$, respectively. We use a batch size of 150, and exclude the normal map information from the first two deconvolution layers. The model is trained during 2,400 epochs, with a total training time of 106 hours. The $\gamma$ parameter is fixed to a constant value of 6, and we perform an annealing of the $\beta$ parameter, which grows linearly from 0 to 2 during the first 1,000 epochs. For the regularization term, we use n=3.

We use RealisticVisionXL4.0 (https://huggingface.co/SG161222/RealVisXL_V4.0) as the base model for our diffusion pipeline. It is a fine-tuning of Stable Diffusion XL, especially aimed at photorealism. Following community recommendations, we train the default architecture of IP-Adapter [ye2023ip] for reconstruction on our training dataset. To ensure accurate appearance embeddings during training, we use the same inputs used to train our appearance encoder, at 512x512 resolution. Using object-centered images with the background masked out, we encourage the learned weights of the adapter to store information aligned with the FactorVAE’s latent space. Additionally, we effectively remove the influence of the text embeddings by using uninformative prompts, such as “image”. Training is done in a NVIDIA A100 GPU with $10^{-4}$ learning rate with batch size of 10. During inference, we use the Diffusers [von-platen-etal-2022-diffusers] implementation of the ControlNet inpainting pipeline for Stable Diffusion XL.

In the main text (Fig. 3), we include the prior traversals plot of our 6D latent space, highlighting its disentanglement and interpretability properties. Further, and thanks to these properties, we study the result of combining, two by two, these disentangled dimensions. Fig. 11 contains the result of combining three different pairs of dimensions. Thanks to the inherent continuity of VAE-based methods, we see how combining the identified dimensions results in progressive changes in the appearance of the Havran geometry.

We include in Fig. 12 additional results of the posterior traversals plot (see main text, Fig. 4, right). Here, we compute posteriors with three additional test samples: a rendered blue rough blob, and two real photographs of a glossy black statue, and a yellow plastic seahorse, respectively. We use the Havran geometry as guidance in the decoder, and highlight in green the samples corresponding to reconstructions of the reference material, akin to what is done in the main text.

As discussed in the main text (Sec. 4.2), our VAE-based appearance encoder suffers from a limited capacity to reconstruct geometries very different from those seen during training. This generalizability issue is not an unexpected behavior, but it hinders the applicability of such encoder alone to reconstruct appearance in unknown scenarios. In Fig. 13 we show the result of reconstructing the real images used in the main text (Fig. 4) with different objective geometries: as reference, we use the blob geometry (row two), which is in the training dataset, and then test the reconstructions for three unknown geometries, namely the monster, the cat, and the chair in rows three, four, and five, respectively. Analyzing the results, we see how the model is able to apply the reference appearance to some extent, but struggles in the estimation of illumination cues (i.e. shadows) and high frequency details of the geometries, while generating unexpected artifacts. This issue arises from the use of unknown geometries, which the decoder is not able to properly interpret.

In the main manuscript (Fig. 5), we plotted the evolution of the MIR and MIS metrics when modifying the dimensionality of the latent space. Here, in Fig. 18, we visualize:

• •

  The factors captured in each dimension of the latent space, by computing the prior traversals plots.

• •

  The informativeness of the space, by plotting the dimension-wise KL loss during training. Higher values in this plot represent that the learned distribution is different from the standard normal distribution $N(0,1)$, and thus are storing more information.

Fig. 18 shows how our latent space of 6 dimensions is the model whose space is more visually disentangled and interpretable, while not leaving uninformative dimensions.

During the design of the final model, we tested four commonly used reconstruction losses in autoencoder architectures, namely BCE, Bernoulli, Huber, and L1-smooth [NEURIPS2019_9015]. In the final implementation, we chose to use the L1-smooth function, due to its superior performance in our experiments, as shown in its improved interpretability, measured by MIR, in Table 3). Additionally, the first two reconstruction functions operate at a considerably higher magnitude than the latter two alternatives (Fig. 14), which caused training instabilities.

In the Sec. 3.1 of the main text, we explain that the geometry guidance in the decoder of the FactorVAE-based architecture is implemented by adequately concatenating the channels to layers of the decoder pipeline. To adequately match the shape of the maps to that of the feature maps of the layers, we need to resize the normal map image. Here we ablate the algorithm used to perform such resizing of the normal map, comparing the nearest neighbors (NN) and bilinear methods. Fig. 15 shows the performance of two models, each one trained with one of the alternatives, when reconstructing a test geometry. We can clearly see how the model that uses the bilinear interpolation achieves a better understanding of the unseen geometry, which is reflected in a reduced (but not inexistent) number of artifacts. This is probably due to a smoother reduction of the resolution by the bilinear algorithm, in contrast to the nearest neighbors one.

Once trained, our diffusion pipeline is expected to resemble the good properties of our appearance encoder (e.g., disentanglement, interpretability, continuity). To evaluate this, we create a version of the posterior traversals plot (main text, Fig. 4, right) with the designed diffusion-based pipeline, using the four dimensions that contain appearance-related features. We use the remaining two dimensions encoding illumination features exclusively to guide the pipeline on the desired lighting conditions, thus we do not perform a traversal on such dimensions. For it, we start from the setup displayed in the teaser (main text, Fig. 1, left), where we apply a rough purple material to the painting of a car. Starting from this appearance transfer (figures marked in green), we systematically vary each of the four dimensions, obtaining as a result the visualization of Fig. 20. Rows follow the same order as the traversals in Fig. 12, namely lightness, hue #1, hue #2, and gloss. Note the overall smoothness of transitions, achieved thanks to the guidance of our custom appearance encoder. When traversing the same dimension as the appearance transfer (e.g. trying to make the paint even more purple), the representation may overflow the ranges seen during training, creating saturated results. To further evaluate the robustness of our method, we generate diffusion-based posterior traversals using 15 combinations of the three geometries and five appearances illustrated in Fig. 16. The resulting traversals are presented in Figs. 20- 34. These results encompass a diverse range of appearances and reference geometries, showcasing our method’s applicability to real-world scenarios, as well as its limitations in some challenging examples.

As described in the main paper, our inference pipeline leverages a combination of two ControlNets [zhang2023adding] to provide geometry guidance. Each ControlNet must be appropriately weighted, as these weights directly influence the final image. In our work, we use fixed weights of 0.2 for the Depth ControlNet and 0.9 for the Canny ControlNet. To analyze the impact of these weights, we run an ablation study, presented in Fig. 17. As observed, the Canny map may contain some ambiguous edges that require complex interpretation during inference. When relying solely on this information (D=0.0, C=1.0), the pipeline may misinterpret certain edges as being outside of the main geometry, leading to artifacts such as the black blob (first row, second column). Conversely, the depth map helps to better identify the geometry but lacks some details of the scene. If only conditioning the diffusion pipeline with the estimated depth map (Fig. 17, last row), high-frequency details are significantly diminished, which is especially noticeable in the sphere (last row, third column). Among the tested configurations, our chosen weights offer the best balance between preserving geometric structure and maintaining high-frequency fine details.