GStreamer笔记五: Media Formats and Pad Capabilities

此次所述有关pad功能。关于pad，前面已有简单介绍，pad功能是GStreamer的一个基本元素，由于框架会自动处理他们，所以大多数时候它们是不可见的。本次主要了解的是关于pad功能的检索。

基本概念

Pads

之前已经说过，pad相当于一个接口，允许信息通过或者离开一个元素。Pad的功能(Capabilities或者简写为Cap)指定哪种信息可以通过pad。例如“分辨率为320x200像素，每秒30帧的RGB视频”，或“每采样音频16位，每秒44100采样率的5.1声道”，甚至压缩格式如mp3或h264等。 Pad可以支持多种功能(如一个video sink可以支持不同类型的RGB或者YUV格式的视频)，而且功能也可以指定范围(如一个audio sink可以支持每秒1-48000个采样的采样率)。但从pad到pad的实际信息必须仅有一个明确指定的类型。两个相连的pad通过一个协商过程达到一个共同的类型，从而使其pad功能变得固定(仅有一种类型，不包含范围)。
要让两个元素连接在一起，他们的能力必须有一个公共子集，否则不可能相互理解。这也是能力的主要目标。
应用程序开发人员通常会通过将元素连接到一起来构建管道（如果使用playbin之类的全部元素，则程度较低）。在这种情况下需要知道元素的Pad Caps，或者至少知道当GStreamer拒绝连接两个协商错误的元素时它们的Caps是什么。

Pad templates

Pad是从pad模板创建的，pad模板表示了pad可能具有的所有功能。模板可用于创建一些相似的pad，并且允许早期拒绝元素之间的连接：如果其pad模板的功能没有公共子集（相交为空），则无需进一步协商。
pad模板可视为协商过程第一步。随着过程的演变，实际的Pads被实例化并且其能力被提炼，直到它们被固定（或者协商失败）。

Capabilities examples

一个能力的例子如下:

SINK template: 'sink'
  Availability: Always
  Capabilities:
    audio/x-raw
               format: S16LE
                 rate: [ 1, 2147483647 ]
             channels: [ 1, 2 ]
    audio/x-raw
               format: U8
                 rate: [ 1, 2147483647 ]
             channels: [ 1, 2 ]

如代码中所示，该pad是一个接收端(sink)，且一直可用。它支持两种媒体，包括整数格式的原始音频（audio/x-raw）：带符号的16位小端 ^[1]和无符号8位。方括号表示一个范围：例如，通道数从1到2不等。

SRC template: 'src'
  Availability: Always
  Capabilities:
    video/x-raw
                width: [ 1, 2147483647 ]
               height: [ 1, 2147483647 ]
            framerate: [ 0/1, 2147483647/1 ]
               format: { I420, NV12, NV21, YV12, YUY2, Y42B, Y444, YUV9, YVU9, Y41B, Y800, Y8, GREY, Y16 , UYVY, YVYU, IYU1, v308, AYUV, A420 }

video/x-raw表示这个信号源输出原始视频。它支持多种尺寸和帧率，以及一组YUV格式（大括号列表表示）。所有这些格式都表示图像平面的不同的填充和下采样。

Last remarks

可以使用gst-inspect-1.0工具了解所有GStreamer元素的Caps。
一些元素查询底层硬件支持的格式，并相应地提供他们的pad功能（通常进入READY或更高状态时在这样做）。因此，显示的能力可能因平台而异，也可能从一次执行到下一次执行就变了（即使这种情况很少）。
本次将会实例化两个元素（这次通过其工厂函数），显示其pad模板，连接它们并设置管道播放。在每次状态改变时，sink元素的Pad的功能会被显示，所以可以观察到协商如何进行直到Pad Caps固定。

Pad功能实例

一个普通的有关pad caps的例子如下:

cpp

#include <gst/gst.h>

/* Functions below print the Capabilities in a human-friendly format */
static gboolean print_field (GQuark field, const GValue * value, gpointer pfx) {
  gchar *str = gst_value_serialize (value);

  g_print ("%s  %15s: %s\n", (gchar *) pfx, g_quark_to_string (field), str);
  g_free (str);
  return TRUE;
}

static void print_caps (const GstCaps * caps, const gchar * pfx) {
  guint i;

  g_return_if_fail (caps != NULL);

  if (gst_caps_is_any (caps)) {
    g_print ("%sANY\n", pfx);
    return;
  }
  if (gst_caps_is_empty (caps)) {
    g_print ("%sEMPTY\n", pfx);
    return;
  }

  for (i = 0; i < gst_caps_get_size (caps); i++) {
    GstStructure *structure = gst_caps_get_structure (caps, i);

    g_print ("%s%s\n", pfx, gst_structure_get_name (structure));
    gst_structure_foreach (structure, print_field, (gpointer) pfx);
  }
}

/* Prints information about a Pad Template, including its Capabilities */
static void print_pad_templates_information (GstElementFactory * factory) {
  const GList *pads;
  GstStaticPadTemplate *padtemplate;

  g_print ("Pad Templates for %s:\n", gst_element_factory_get_longname (factory));
  if (!gst_element_factory_get_num_pad_templates (factory)) {
    g_print ("  none\n");
    return;
  }

  pads = gst_element_factory_get_static_pad_templates (factory);
  while (pads) {
    padtemplate = pads->data;
    pads = g_list_next (pads);

    if (padtemplate->direction == GST_PAD_SRC)
      g_print ("  SRC template: '%s'\n", padtemplate->name_template);
    else if (padtemplate->direction == GST_PAD_SINK)
      g_print ("  SINK template: '%s'\n", padtemplate->name_template);
    else
      g_print ("  UNKNOWN!!! template: '%s'\n", padtemplate->name_template);

    if (padtemplate->presence == GST_PAD_ALWAYS)
      g_print ("    Availability: Always\n");
    else if (padtemplate->presence == GST_PAD_SOMETIMES)
      g_print ("    Availability: Sometimes\n");
    else if (padtemplate->presence == GST_PAD_REQUEST) {
      g_print ("    Availability: On request\n");
    } else
      g_print ("    Availability: UNKNOWN!!!\n");

    if (padtemplate->static_caps.string) {
      GstCaps *caps;
      g_print ("    Capabilities:\n");
      caps = gst_static_caps_get (&padtemplate->static_caps);
      print_caps (caps, "      ");
      gst_caps_unref (caps);

    }

    g_print ("\n");
  }
}

/* Shows the CURRENT capabilities of the requested pad in the given element */
static void print_pad_capabilities (GstElement *element, gchar *pad_name) {
  GstPad *pad = NULL;
  GstCaps *caps = NULL;

  /* Retrieve pad */
  pad = gst_element_get_static_pad (element, pad_name);
  if (!pad) {
    g_printerr ("Could not retrieve pad '%s'\n", pad_name);
    return;
  }

  /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */
  caps = gst_pad_get_current_caps (pad);
  if (!caps)
    caps = gst_pad_query_caps (pad, NULL);

  /* Print and free */
  g_print ("Caps for the %s pad:\n", pad_name);
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  gst_object_unref (pad);
}

int main(int argc, char *argv[]) {
  GstElement *pipeline, *source, *sink;
  GstElementFactory *source_factory, *sink_factory;
  GstBus *bus;
  GstMessage *msg;
  GstStateChangeReturn ret;
  gboolean terminate = FALSE;

  /* Initialize GStreamer */
  gst_init (&argc, &argv);

  /* Create the element factories */
  source_factory = gst_element_factory_find ("audiotestsrc");
  sink_factory = gst_element_factory_find ("autoaudiosink");
  if (!source_factory || !sink_factory) {
    g_printerr ("Not all element factories could be created.\n");
    return -1;
  }

  /* Print information about the pad templates of these factories */
  print_pad_templates_information (source_factory);
  print_pad_templates_information (sink_factory);

  /* Ask the factories to instantiate actual elements */
  source = gst_element_factory_create (source_factory, "source");
  sink = gst_element_factory_create (sink_factory, "sink");

  /* Create the empty pipeline */
  pipeline = gst_pipeline_new ("test-pipeline");

  if (!pipeline || !source || !sink) {
    g_printerr ("Not all elements could be created.\n");
    return -1;
  }

  /* Build the pipeline */
  gst_bin_add_many (GST_BIN (pipeline), source, sink, NULL);
  if (gst_element_link (source, sink) != TRUE) {
    g_printerr ("Elements could not be linked.\n");
    gst_object_unref (pipeline);
    return -1;
  }

  /* Print initial negotiated caps (in NULL state) */
  g_print ("In NULL state:\n");
  print_pad_capabilities (sink, "sink");

  /* Start playing */
  ret = gst_element_set_state (pipeline, GST_STATE_PLAYING);
  if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state (check the bus for error messages).\n");
  }

  /* Wait until error, EOS or State Change */
  bus = gst_element_get_bus (pipeline);
  do {
    msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS |
        GST_MESSAGE_STATE_CHANGED);

    /* Parse message */
    if (msg != NULL) {
      GError *err;
      gchar *debug_info;

      switch (GST_MESSAGE_TYPE (msg)) {
        case GST_MESSAGE_ERROR:
          gst_message_parse_error (msg, &err, &debug_info);
          g_printerr ("Error received from element %s: %s\n", GST_OBJECT_NAME (msg->src), err->message);
          g_printerr ("Debugging information: %s\n", debug_info ? debug_info : "none");
          g_clear_error (&err);
          g_free (debug_info);
          terminate = TRUE;
          break;
        case GST_MESSAGE_EOS:
          g_print ("End-Of-Stream reached.\n");
          terminate = TRUE;
          break;
        case GST_MESSAGE_STATE_CHANGED:
          /* We are only interested in state-changed messages from the pipeline */
          if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) {
            GstState old_state, new_state, pending_state;
            gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
            g_print ("\nPipeline state changed from %s to %s:\n",
                gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
            /* Print the current capabilities of the sink element */
            print_pad_capabilities (sink, "sink");
          }
          break;
        default:
          /* We should not reach here because we only asked for ERRORs, EOS and STATE_CHANGED */
          g_printerr ("Unexpected message received.\n");
          break;
      }
      gst_message_unref (msg);
    }
  } while (!terminate);

  /* Free resources */
  gst_object_unref (bus);
  gst_element_set_state (pipeline, GST_STATE_NULL);
  gst_object_unref (pipeline);
  gst_object_unref (source_factory);
  gst_object_unref (sink_factory);
  return 0;
}

所需要的库只有gstreamer-1.0。

代码分析

print_field, print_caps和print_pad_templates简单的以友好的格式显示了caps的结构，更多的GstCaps内部结构组织可以阅读GStreamer的关于Pad Caps的文档。

cpp

/* Shows the CURRENT capabilities of the requested pad in the given element */
static void print_pad_capabilities (GstElement *element, gchar *pad_name) {
  GstPad *pad = NULL;
  GstCaps *caps = NULL;

  /* Retrieve pad */
  pad = gst_element_get_static_pad (element, pad_name);
  if (!pad) {
    g_printerr ("Could not retrieve pad '%s'\n", pad_name);
    return;
  }

  /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */
  caps = gst_pad_get_current_caps (pad);
  if (!caps)
    caps = gst_pad_query_caps (pad, NULL);

  /* Print and free */
  g_print ("Caps for the %s pad:\n", pad_name);
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  gst_object_unref (pad);
}

gst_element_get_static_pad函数从给定的元素中检索命名的pad。这个pad是静态的，因为它总是存在于元素中。有关Pad可用性的更多信息，可以阅读有关Pads的GStreamer文档。
然后调用gst_pad_get_current_caps来检索pad的当前caps，它可以是固定的或不固定的，具体取决于协商过程的状态。甚至可能不存在，这种情况下调用gst_pad_query_caps来检索当前可接受的pad caps。当前可接受的caps将成为处于NULL状态的pad模板的caps，但可能会在以后的状态中更改，因为可能会查询实际的硬件功能。
然后将这些功能打印出来:

/* Create the element factories */
source_factory = gst_element_factory_find ("audiotestsrc");
sink_factory = gst_element_factory_find ("autoaudiosink");
if (!source_factory || !sink_factory) {
  g_printerr ("Not all element factories could be created.\n");
  return -1;
}

/* Print information about the pad templates of these factories */
print_pad_templates_information (source_factory);
print_pad_templates_information (sink_factory);

/* Ask the factories to instantiate actual elements */
source = gst_element_factory_create (source_factory, "source");
sink = gst_element_factory_create (sink_factory, "sink");

之前直接使用gst_element_factory_make函数创建了元素，并跳过了关于工厂的讨论。GstElementFactory负责实例化由其工厂名称标识的特定类型的元素。
可以使用gst_element_factory_find来创建一个类型为“videotestsrc”的工厂，然后通过gst_element_factory_create使用它实例化多个“videotestsrc”元素。gst_element_factory_make实际上是gst_element_factory_find+gst_element_factory_create的快捷方式。
Pad模板已经可以通过工厂来访问，所以一旦工厂被创建，它们就会被打印出来。
跳过管道的创建以及开始，转到State-Changed消息处理：

cpp

case GST_MESSAGE_STATE_CHANGED:
  /* We are only interested in state-changed messages from the pipeline */
  if (GST_MESSAGE_SRC (msg) == GST_OBJECT (pipeline)) {
    GstState old_state, new_state, pending_state;
    gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
    g_print ("\nPipeline state changed from %s to %s:\n",
        gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
    /* Print the current capabilities of the sink element */
    print_pad_capabilities (sink, "sink");
  }
  break;

该段代码在每次Pipeline状态发生改变时简单的打印出当前的pad caps。可以在输出中看到初始caps（pad模板的caps）是如何逐步完善直至完全固定（包含无范围单一类型）。

对于整型/长整型等数据类型，小端与大端相反，按照从低地址到高地址的顺序存放数据的高位字节到低位字节。 ↩︎