How to recover a docker image from Kubernetes Node

When you’ve lost a Docker image from your remote repository but it’s still cached on a Kubernetes Node, you can recover it using containerd’s command-line tool, ctr. This tool is typically bundled with containerd installations.

Using ctr

ctr, often included in containerd installations, can be used in conjunction with crictl, a standalone kubernetes-sigs project. Here’s how to interact with ctr and containerd:

  1. Set up the environment: To interact with ctr define this environment variable:

    export CONTAINER_RUNTIME_ENDPOINT="unix:///run/containerd/containerd.sock"

  2. For basic authentication (e.g., with AWS ECR):

     ECR_REGION=us-east-1
     ECR_PASSWORD=$(aws ecr get-login-password --region $ECR_REGION)
    
  3. Tagging and pushing images:

     ctr -n k8s.io image tag <IMAGE> <ECR:IMAGE:TAG>
     ctr -n k8s.io image push --user "AWS:$ECR_PASSWORD" <ECR:IMAGE:TAG>
    

    Note: The ‘k8s.io’ namespace is required for image interactions.

Other Helpful Tools

  • nerdctrl: Docker-compatible CLI for containerd
  • crictl: crictl provides a CLI for CRI-compatible container runtimes.

How To Run Daemonset Pods In Certain Nodes

In scenarios that you want to run a Daemonset, but want to run daemonset pods in specific nodes or avoid running pods without deleting the daemonset for a particular timeframe. You can use labes with nodeSelector.

The nature of a DaemonSet was designed to run one pod per node in your Kubernetes cluster, if you have 3 nodes, you will see 3 daemonset pods running.

To effectively reduce the number of Daemonset pods to 0 without deleting the DaemonSet or to chose which node will be running the daemonset pods, it is recommended to use Node Selectors. Here are two ways to implement this solution:

Setting Labels to Kubernetes Nodes

  1. Add a label to all existing nodes. In this example I will use “fluent-bit=false” to control how many FluentBit Daemonset pods will be running in my nodes. To add a label use this command:

    kubectl get nodes -o name | xargs -I{} kubectl label {} fluent-bit=false --overwrite

    Note: You may need to rerun this command if new nodes are added to the cluster.

  2. Verify the label change:

    kubectl get nodes --show-labels

  3. Modify your DaemonSet manifest adding a new selector:

     nodeSelector:
     kubernetes.io/os: linux
     fluent-bit: "true"
    

    For example:

     apiVersion: apps/v1
     kind: DaemonSet
     metadata:
       name: fluent-bit
       namespace: amazon-cloudwatch
       labels:
         k8s-app: fluent-bit
         version: v1
         kubernetes.io/cluster-service: "true"
     spec:
       selector:
         matchLabels:
           k8s-app: fluent-bit
       template:
         metadata:
           labels:
             k8s-app: fluent-bit
             version: v1
             kubernetes.io/cluster-service: "true"
         spec:
           containers:
             - name: fluent-bit
               image: public.ecr.aws/aws-observability/aws-for-fluent-bit:2.32.4
               imagePullPolicy: Always
           nodeSelector:
             kubernetes.io/os: linux
             fluent-bit: "true"
    
  4. Apply the updated configuration:

    kubectl apply -f <manifest-name>.yaml

  5. Verify that your Daemonset pods are not running.
  6. To re-enable Daemonset pods in the future, you can update the node labels to a desired label and value. e.g. “fluent-bit=true”:

Patching Node Selectors

Patching on the fly. In this case, the command is setting the nodeSelector to include a label “non-existing”: “true”, which means that the fluent-bit pods will only be scheduled on nodes that have this label.

kubectl -n amazon-cloudwatch patch daemonset fluent-bit -p '{
  "spec": {
    "template": {
      "spec": {
        "nodeSelector": {
          "non-existing": "true"
        }
      }
    }
  }
}'

How to Configure LogMonitor and CloudWatch Logs for ECS Windows Tasks

Windows applications often use different logging mechanisms than their Linux counterparts. Instead of relying on STDOUT, Windows apps typically utilize Windows-specific logging methods, such as ETW, the Event Log, or custom log files like IIS logs. If you’re running a Windows container in an ECS cluster and want to capture Windows and IIS logs in CloudWatch, you’ll need to implement Log Monitor instrumentation. This setup redirects IIS and system logs to STDOUT, allowing the awslogs driver to automatically capture and forward them to CloudWatch Logs.

Setting Up Log Monitor and CloudWatch for IIS Logs

Follow these steps to configure Log Monitor and send IIS logs to CloudWatch:

  1. Identify the Log Providers.

    Determine the providers to include in the configuration file using:

    logman query providers | findstr "<GUID or Provider Name>"

    For IIS, you can use:

    IIS: WWW Server with GUID 3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83

  2. Create the LogMonitorConfig.json file.

    This file specifies which logs to capture. Below is an example configuration capturing system logs, application logs, and IIS logs:

     {
       "LogConfig": {
         "sources": [
           {
             "type": "EventLog",
             "startAtOldestRecord": true,
             "eventFormatMultiLine": false,
             "channels": [
               {
                 "name": "system",
                 "level": "Information"
               },
               {
                 "name": "application",
                 "level": "Error"
               }
             ]
           },
           {
             "type": "File",
             "directory": "c:\\inetpub\\logs",
             "filter": "*.log",
             "includeSubdirectories": true,
             "includeFileNames": false
           },
           {
             "type": "ETW",
             "eventFormatMultiLine": false,
             "providers": [
               {
                 "providerName": "IIS: WWW Server",
                 "providerGuid": "3A2A4E84-4C21-4981-AE10-3FDA0D9B0F83",
                 "level": "Information"
               },
               {
                 "providerName": "Microsoft-Windows-IIS-Logging",
                 "providerGuid": "7E8AD27F-B271-4EA2-A783-A47BDE29143B",
                 "level": "Information"
               }
             ]
           }
         ]
       }
     }
    
  3. Download Log Monitor

    Download LogMonitor.exe from the GitHub repository binaries.

  4. Update Your Dockerfile

    Integrate Log Monitor into your Docker container. Refer to the usage guide here.

    Example Dockerfile:

     FROM mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2022
    
     WORKDIR /LogMonitor
    
     COPY LogMonitorConfig.json LogMonitor.exe ./
    
     ENTRYPOINT ["C:\\LogMonitor\\LogMonitor.exe", "C:\\ServiceMonitor.exe", "w3svc"]
    
  5. Configure CloudWatch Logs in the Task Definition

    Enable CloudWatch logs and set up the log driver in your ECS task definition:

    Task definition example:

     "logConfiguration": {
         "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/iis-logs",
             "awslogs-create-group": "true",
             "awslogs-region": "us-east-1",
             "awslogs-stream-prefix": "ecs"
          }
       }
    
  6. Rebuild and Deploy

    • Rebuild the container image and push it to your ECR repository.
    • Update the ECS task definition to use the new image and deploy it to the cluster.

Viewing Logs in CloudWatch

  1. Open the Amazon ECS console.
  2. Navigate to the cluster containing the task.
  3. Select the Tasks tab and choose a task to view.
  4. Expand the container details by clicking the arrow next to its name.
  5. In the Log Configuration section, select View logs in CloudWatch to access the log stream.

Example Outputs

  • ETW and Log Entries:

    cloudwatch

  • Application Exceptions:

    applicationerror

Building a custom API Gateway with YARP

Whenever you are designing an architecture with microservices, you might encounter in how to implement an API Gateway, since you need a way to communicate and consume multiple services, generally through APIs. A possible solution is to have a single entry point for all your clients and implement an API Gateway, which will handle all the requests and route those to appropiate microservices.

There are different ways to implement an API Gateway or pay for built-in services in cloud hostings.

In this post I will pick the easiest way that I found to create one for a microservice architecture using .NET and YARP. Here is a general overview of a microservice architecture.

architecture

YARP

YARP (which stands for “Yet Another Reverse Proxy”) is an open-source project that Microsoft built for improving routing through internal services using a built-in reverse proxy server. This become very popular and was implemented for several Azure products as App Service.

  • To get started, you need to create an ASP.NET core empty project. I chose .NET 7.0 for this post.
dotnet new web -n ApiGateway -f net7.0
Install-Package Yarp.ReverseProxy
<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Yarp.ReverseProxy" Version="2.0.1" />
  </ItemGroup>

</Project>
  • Add the YARP middleware to your Program.cs.
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));
var app = builder.Build();
app.MapReverseProxy();
app.Run();
  • To add the YARP configuration you will use appsettings.json file. YARP uses routes and clusters, regularilly inside ReverseProxy object defined in builder configuration section. You can find more information about different configuration settings here.

  • In this example, I am using Products and Employee microservices. So I will have routes like employee-route and product-route and clusters as product-cluster and employee-cluster pointing to destinations. Open your appsettings.json and apply the following configuration.

{
  "ReverseProxy": {
    "Routes": {
      "product-route": {
        "ClusterId": "product-cluster",
        "Match": {
          "Path": "/p/{**catch-all}",
          "Hosts": ["*"]
        },
        "Transforms": [
          {
            "PathPattern": "{**catch-all}"
          }
        ]
      },
      "employee-route": {
        "ClusterId": "employee-cluster",
        "Match": {
          "Path": "/e/{**catch-all}",
          "Hosts": ["*"]
        },
        "Transforms": [
          {
            "PathPattern": "{**catch-all}"
          }
        ]
      }
    },
    "Clusters": {
      "product-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "http://localhost:3500/v1.0/invoke/product-api/method/"
          }
        }
      },
      "employee-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "http://localhost:3500/v1.0/invoke/employee-api/method/"
          }
        }
      }
    }
  }
}
  • In scenarios that you need to allow CORS to specific origins you can add a cors policy described in this Microsoft Doc. Here is a configuration example:
var builder = WebApplication.CreateBuilder(args);
var allowOrigins = "_allowOrigins";
builder.Services.AddCors(options =>
{
    options.AddPolicy(allowOrigins, policy =>
    {
        policy
          .WithOrigins("http://localhost:3000", "http://127.0.0.1")
          .SetIsOriginAllowedToAllowWildcardSubdomains()
          .AllowAnyHeader()
          .WithMethods("GET", "POST", "PUT", "DELETE")
          .AllowCredentials();
    });
});
var app = builder.Build();
app.UseCors(allowOrigins);
  • Then add this CORs policy inside Routes as followed:
"Routes": {
      "product-route": {
        "ClusterId": "product-cluster",
        "CorsPolicy": "_allowOrigins",
        "Match": { "..." },
        "Transforms": [ "..."]
      },
}
  • Finally if you want to get more information about YARP logging for future debugging or production information, you can add the YARP log level (Information,Warning or Error) inside Logging object as followed:
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Yarp": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  }
  • Run the application with dotnet run and use Postman or curl to test the endpoints defined in your paths e.g. http://localhost:5212/p/v1/.

    Check all possible configurations and transformations in the YARP documentation.