Graphics Overview ================= In this chapter we present Linux's display model. Generalities ------------ From a programmer's point of view, screens basically display rectangular grids of sample points (or "pixels" standing for "picture elements") where each point can have a different color. The display resolution is the number of pixels in each dimension: for example, a display resolution of 1920x1080 means that there are 1920 pixels in the horizontal dimension and 1080 in the vertical dimension. On some hardware, we may also use `sub-pixel rendering `_: pixel colors are generated by blending several sub-pixel colors. If we know the physical layout of these sub-pixels, we may want to precisely select the sub-pixel colors to enhance the overall output. There can be a lot of different hardware configurations. It is common to have computers with multiple connected display devices: multi-screens, video-projector, TV, etc. Each display supports its own modes of operation (display resolution, refresh rates, etc.). Besides, we may want each device to display something different or some of them to display the same thing ("clone" mode): in the latter case, what if the display resolutions or the other mode settings are different? It is also possible to have several graphic cards in the same computer: * each card can provide its own set of output display ports; * several cards may be physically connected together to accelerate the display on one of their display ports; * several cards may be connected to the same output connectors and the system can switch between the different graphic cards. Usually there are two cards: a power-consuming performant one and a less power-consuming basic one used to save power. In this chapter we present a simple approach to display rendering: basically the picture to display is generated by the CPU in host memory and then transferred to the GPU memory (implicitly by using memory mapping). Most recent graphic cards propose more efficient approaches: the picture to display is generated and transformed directly by the graphic card. Instead of sending a picture, the host sends commands or programs to be executed by the GPU. Currently Linux doesn't propose a unified interface to advanced graphic card capabilities from different vendors (these are usually handled by MESA in user-space and accessed through the OpenGL interface). ``haskus-system`` doesn't provide support for them yet. In this chapter, we describe the Kernel Mode Setting (KMS) and the Direct Rendering Manager (DRM) interfaces. In usual Linux distributions, some graphic card manufacturers provide closed-source proprietary drivers that do not support theses interfaces: they use a kernel module and user-space libraries that communicate together by using a private protocol. The user-space libraries provide implementations of standard high-level interfaces such as OpenGL and can be used by rendering managers such as X.org. ``haskus-system`` doesn't offer a way to use these drivers. Display Model ------------- Linux's display model is composed of several entities that interact with each other. These entities are represented on the following graph: .. image:: /_static/images/system/graphics_entities.svg :class: img_center Numbers indicate relationship arities: for instance a ``Controller`` can be connected to at most a single ``Framebuffer``, but a ``Framebuffer`` can be used by a variable amount of ``Controllers``. The arrows indicate entities that store references to others entities (at the tip of the arrow). In order to display something, we have to configure a pipeline that goes from some ``surfaces`` (pixel data stored in a memory buffers) to some ``connectors`` (entity representing physical ports onto which display devices are connected). Card ~~~~ The following schema shows a contrived example of a graphic card containing some of these entities. .. image:: /_static/images/system/graphics_card.svg :class: img_center Entities belong to a single graphic card: they can't be shared between several graphic cards (if your system has more than one of these). Connectors ~~~~~~~~~~ Each physical port where you can plug a display device (a monitor, a video-projector, etc.) corresponds to a ``Connector`` entity in the display model. The following code shows how to retrieve the graphic card objects and how to display information about each connector: .. code:: haskell import Haskus.System import Haskus.Arch.Linux.Graphics.State main :: IO () main = runSys <| do sys <- defaultSystemInit term <- defaultTerminal -- get graphic card devices cards <- loadGraphicCards (systemDeviceManager sys) forM_ cards <| \card -> do state <- readGraphicsState (graphicCardHandle card) >..~!!> assertShow "Cannot read graphics state" -- get connector state and info let conns = graphicsConnectors state -- show connector state and info writeStrLn term (show conns) void powerOff When executed in ``QEMU``, this code produces the following output: .. code:: haskell -- Formatting has been enhanced for readability [ Connector { connectorID = ConnectorID 21 , connectorType = Virtual , connectorByTypeIndex = 1 , connectorState = Connected (ConnectedDevice { connectedDeviceModes = [ Mode { ... , modeClock = 65000 , modeHorizontalDisplay = 1024 , modeVerticalDisplay = 768 , modeVerticalRefresh = 60 , modeFlags = fromList [ModeFlagNHSync,ModeFlagNVSync] , modeStereo3D = Stereo3DNone , modeType = fromList [ModeTypePreferred,ModeTypeDriver] , modeName = "1024x768" } , ... ] , connectedDeviceWidth = 0 , connectedDeviceHeight = 0 , connectedDeviceSubPixel = SubPixelUnknown , connectedDeviceProperties = [ Property { propertyMeta = PropertyMeta { ... , propertyName = "DPMS" , propertyType = PropEnum [ (0,"On") , (1,"Standby") , (2,"Suspend") , (3,"Off")] } , propertyValue = 0 } ] }) , connectorPossibleEncoderIDs = [EncoderID 20] , connectorEncoderID = Just (EncoderID 20) , connectorHandle = Handle ... } ] Each connector reports its type in the ``connectorType`` field: in our example it is a virtual port because we use ``QEMU``, but it could have been ``VGA``, ``HDMI``, ``TV``, ``LVDS``, etc. If there are several connectors of the same type in the same card, you can distinguish them with the ``connectorByTypeIndex`` field. You can check that a display device is actually plugged in a connector with the ``connectorState`` property: in our example, there is a (virtual) screen connected. We can get more information about the connected device: * ``connectedDeviceModes``: modes supported by the connected display device. In particular, a display resolution is associated to each mode. In our example, the display resolution of the first mode is 1024x768; the other modes have been left out for clarity. * ``connectedDeviceWidth`` and ``connectedDeviceHeight``: some display devices report their physical dimensions in millimeters. * ``connectedDeviceSubPixel``: whether the device uses some kind of sub-pixel technology. * ``connectedDeviceProperties``: device specific properties. In this example, there is only a single property named "DPMS" which can take 4 different values ("On", "Standby", "Suspend", "Off") and whose current value is 0 ("On"): this property can be used to switch the power mode of the screen. A ``connector`` gets the data to display from an ``encoder``: * ``connectorPossibleEncoderIDs``: list of encoders that can be used as sources. * ``connectorEncoderID``: identifier of the currently connected encoder, if any. Detecting Plugging/Unplugging ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ We can adapt what our system displays to the connected screens, but how do we detect when a screen is connected or disconnected? A solution would be to periodically check the value of the ``connectorState`` property. But a better method is to use a mechanism explained in the `basic device management `_ page: when the state of a connector changes, the kernel sends to the user-space an event similar to the following one: .. code:: haskell KernelEvent { kernelEventAction = ActionChange , kernelEventDevPath = "/devices/.../drm/card0" , kernelEventSubSystem = "drm" , kernelEventDetails = fromList [("DEVNAME","drm/card0") ,("MAJOR","226") ,("MINOR","0") ,("HOTPLUG","1") ,("SEQNUM","1259")]} When the system receives this event, it knows it has to check the state of the connectors. Note that the number of ``connector`` entities may change dynamically. For instance a single ``DisplayPort`` connector supporting the Multi-Stream Transport (MST) allows several monitors to be connected in sequence (daisy-chaining): each monitor receives its own video stream and appears as a different ``connector`` entity. It is also possible to connect a MST hub that increases the number of ``connector`` entities. Encoders ~~~~~~~~ Encoders convert pixel data into signals expected by connectors: for instance ``DVI`` and ``HDMI`` connectors need a ``TMDS`` encoder. Each card provides a set of encoders and each of them can only work with some controllers and some connectors. There may be a 1-1 relationship between an ``encoder`` and a ``connector``, in which case the link between them should already be set. We can display information about encoders using a code similar to the code above for connectors. When executed into ``QEMU``, we get the following result: .. code:: haskell [ Encoder { encoderID = EncoderID 20 , encoderType = EncoderTypeDAC , encoderControllerID = Just (ControllerID 19) , encoderPossibleControllers = [ControllerID 19] , encoderPossibleClones = [] , encoderHandle = Handle ... } ] As we can observe, the graphic card emulated by ``QEMU`` emulates a single ``DAC`` encoder. The ``encoderPossibleClones`` field contains the sibling encoders that can be used for cloning: only these encoders can share the same controller as a source. Controllers ~~~~~~~~~~~ Controllers let you configure: * The display mode (display resolution, etc.) that will be used by the display devices that are connected to the controller through an encoder and a connector. * The primary source of the pixel data from a ``FrameBuffer`` entity We can display information about controllers using a code similar to the code above for connectors. When executed into ``QEMU``, we get the following result: .. code:: haskell [ Controller { controllerID = ControllerID 19 , controllerMode = Just (Mode { ...}) , controllerFrameBuffer = Just (FrameBufferPos { frameBufferPosID = FrameBufferID 46 , frameBufferPosX = 0 , frameBufferPosY = 0 }) , controllerGammaTableSize = 256 , controllerHandle = Handle ... } ] * ``controllerMode``: the display mode that has to be used by the display device(s). * ``controllerFrameBuffer``: the ``FrameBuffer`` entity used as a data source and the coordinates in the ``FrameBuffer`` contents. Planes ~~~~~~ Some controllers can blend several layers together from different ``FrameBuffer`` entities: these layers are called ``Planes``. Controller support at least a ``primary`` plane and they can support others such as cursor or overlay planes. :: TODO: * List plane resources * primary plane * cursor planes * overlay planes * example Framebuffers And Surfaces ~~~~~~~~~~~~~~~~~~~~~~~~~ Planes take their input data from ``FrameBuffer`` entities. ``FrameBuffer`` entities describe how pixel data are encoded and where to find them in the GPU memory. Some pixel encoding formats require more than one memory buffers (``Surface`` entities) that are combined to obtain final pixel colors. :: TODO: * Pixel formats * FrameBuffer dirty * Mode * Generic buffers * Note on accelerated buffers If we use an unaccellerated method ("dumb buffers" in Linux terminology) where the graphics data are fulling generated by the CPU, applications only have to map the contents of the ``Surface`` entities into their memory address spaces and to modify it to change what is displayed. Further Reading --------------- As explained in the :ref:`device-management` section, device drivers can support the ``ioctl`` system call to handle device specific commands from the user-space. The display interface is almost entirely based on it. Additionally, ``mmap`` is used to map graphic card memory in user-space and ``read`` is used to read events (V-Blank and page-flip asynchronous completion). In usual Linux distributions, the ``libdrm`` library provides an interface over these system calls. You can learn about the low-level interface by reading the ``drm`` manual (``man drm``) or its `source code `_. David Herrmann has written `a good tutorial `_ explaining how to use the legacy low-level display interface in the form of C source files with detailed comments. While some details of the interface have changed since he wrote it (e.g., the way to flip frame buffers and the atomic interface), it is still a valuable source of information. The newer atomic interface is described in an `article `_ `series `_ on LWN called "Atomic mode setting design overview" (August 2015) by Daniel Vetter. `Wayland `_ is the new display system for usual Linux based distributions. It can be a great source of inspiration and of information. You can also read the Linux kernel code located in ``drivers/gpu/drm`` in the kernel sources. Linux multi-GPU: * Buffer sharing is supported with `DRM Prime `_ * GPU switching is supported with `vga_switcheroo `_