n1sdp: update documentation for 2021.10.12 platform release

A new user guide is prepared to provide instructions for bash script
based build setup, the steps to build the firmware, minimal BusyBox
and the Ubuntu distribution with new scripts are captured.

Other documents are updated as applicable for the 2021.10.12 release.

Change-Id: Id103cde03dce2de6b5359b8637fda7a31deced76
Signed-off-by: Khasim Syed Mohammed <khasim.mohammed@arm.com>

docs/n1sdp/Errata-1315703.rst

+*********************************************
+Errata-1315703 WA disabled in Neoverse N1 SDP
+*********************************************
+
+In Trusted Firmware-A, all erratum are disabled by default including the
+1315703. If we want any erratum to be enabled, then we have to explicitly
+enable them in the platform Makefile.
+
+The N1SDP stack disables the workaround for Erratum 1315703 by default,
+so that the N1 CPU performance in N1SDP better reflects that of released
+versions of the N1 for software that does not require mitigation for
+Spectre Variant 4.
+
+N1SDP uses N1 version r1p0, which is affected by Erratum 1315703, which
+is fixed in N1 r3p1. The workaround for r1p0 disables the CPU performance
+feature of bypassing of stores by younger loads. This can significantly
+affect performance. The Erratum is classified "Cat A (Rare)" and requires
+a specific sequence of events to occur.
+
+Disabling this CPU performance feature is also the mitigation for Spectre
+Variant 4 (CVE-2018-3639). On CPUs that provide the PSTATE.SBSS feature,
+the OS selectively applies the mitigation only to programs that require it,
+leaving the performance of other programs unaffected. However, N1 r1p0
+does not have the PSTATE.SBSS feature (which is introduced in N1 r3p1), and
+TF-A does not provide the interface to dynamically disable the CPU
+performance feature. Therefore, applying the workaround penalizes ALL
+software running on N1SDP, including those that do not require the mitigation.
+
+Disabling Errata-1315703 is meant for performance evaluation purposes ONLY
+and should not be used for software that requires a seccomp computing
+environment.
+
+--------------
+
+*Copyright (c) 2019-2021, Arm Limited. All rights reserved.*

docs/n1sdp/LES-PRE-21570v1.0 Neoverse N1 system development platform EULA_windows.txt

+12 March 2019                CONFIDENTIAL           LES-PRE-21570 SP-Version 1.0
+
+END USER LICENCE AGREEMENT FOR THE SOFTWARE ASSOCIATED WITH THE
+ARM® NEOVERSE™ N1 SYSTEM DEVELOPMENT PLATFORM
+
+THIS END USER LICENCE AGREEMENT (“LICENCE”) IS A LEGAL AGREEMENT BETWEEN YOU
+(EITHER A SINGLE INDIVIDUAL, OR SINGLE LEGAL ENTITY) AND ARM LIMITED ("ARM")
+FOR THE USE OF THE DELIVERABLES ACCOMPANYING THIS LICENCE. ARM IS ONLY
+WILLING TO LICENSE THE DELIVERABLES TO YOU ON CONDITION THAT YOU ACCEPT ALL
+OF THE TERMS IN THIS LICENCE. BY CLICKING “I AGREE” OR BY INSTALLING OR
+OTHERWISE USING OR COPYING THE DELIVERABLES YOU INDICATE THAT YOU AGREE TO BE
+BOUND BY ALL THE TERMS OF THIS LICENCE. IF YOU DO NOT AGREE TO THE TERMS OF
+THIS LICENCE, ARM IS UNWILLING TO LICENSE THE DELIVERABLES TO YOU AND YOU MAY
+NOT INSTALL, USE OR COPY THE DELIVERABLES, BUT YOU SHOULD PROMPTLY RETURN THE
+DELIVERABLES TO YOUR SUPPLIER AND ASK FOR A REFUND OF ANY LICENCE FEE PAID.
+
+“Hardware” means the Arm® NeoverseTM N1 System Development Platform, which is
+comprised of a hardware development board purchased or borrowed directly from
+Arm or its authorised distributors.
+
+"Deliverables" means any software, firmware, boardfiles, data and
+documentation accompanying this Licence, any printed, electronic or online
+documentation supplied with it, and any updates, patches and modifications
+Arm may make available to you under the terms of this Licence, in all cases
+relating to the supporting deliverables for the Hardware.
+
+“Purpose” means the use of the Deliverables solely for the purposes of: (1)
+your internal development, testing and debugging of software applications
+that are designed to run solely on microprocessors manufactured under licence
+from Arm; and (2) prototyping hardware designs incorporating the Deliverables
+
+
+1.  LICENCE GRANTS.
+DELIVERABLES: Arm hereby grants to you, subject to the terms and conditions
+of this Licence, a non-exclusive, non-transferable licence solely for use on
+the Hardware and only for the Purpose to use and copy the Deliverables
+identified in the Schedule.
+
+You shall not modify the Deliverables. You shall not redistribute or
+sub-licence any of the Deliverables.
+
+2.  RESTRICTIONS ON USE OF THE DELIVERABLES.
+COPYING: You shall not use or copy the Deliverables except as expressly
+authorised in this Licence. You may make one additional copy of the delivered
+Deliverables media or image for backup or archival purposes.
+
+PERMITTED USERS: The Deliverables shall be used only by your employees, or by
+your bona fide sub-contractors for whose acts and omissions you hereby agree
+to be responsible to Arm to the same extent as you are for any acts and
+omissions of your employees, and provided always that such sub-contractors;
+(i) work only onsite at your premises; (ii) comply with the terms of this
+Licence; (iii) are contractually obligated to use the Deliverables only for
+your benefit, and (iv) agree to assign all their work product and any rights
+they create therein in the supply of such work to you. Only the single
+individual, company or other legal entity to whom Arm is supplying this
+Licence may use the Deliverables. Except as provided in this clause, you
+shall not allow third parties (including but not limited to any subsidiary,
+parent or affiliated companies, or offsite contractors you may have) to use
+the Deliverables unless Arm specifically agrees otherwise with you on a case
+by case basis.
+
+NO REMOTE USE: The Deliverables shall only be used onsite at your premises
+and only for your benefit.
+
+MULTIPLE VERSIONS: The media on which the Deliverables resides may contain
+more than one version of the Deliverables, each of which is compatible with a
+different operating system (such as Microsoft Windows and Red Hat Linux).
+
+ACADEMIC OR EDUCATIONAL USERS ONLY: If you or your employer or institution
+paid academic or educational pricing for the Deliverables, or the
+Deliverables are identified as an academic or educational version (together
+“Academic Software”), then notwithstanding anything else in this Licence, YOU
+AGREE TO USE THE ACADEMIC SOFTWARE ONLY FOR ACADEMIC, NON-COMMERCIAL
+PURPOSES, AND ARM DOES NOT GRANT YOU ANY RIGHTS TO DISTRIBUTE OR SUB-LICENSE
+ANY APPLICATIONS DEVELOPED USING THE ACADEMIC SOFTWARE UNDER THIS LICENCE.
+
+REVERSE ENGINEERING: Except to the extent that such activity is permitted by
+applicable law you shall not reverse engineer, decompile or disassemble any
+of the Deliverables. If the Deliverables were provided to you in Europe you
+shall not reverse engineer, decompile or disassemble any of the Deliverables
+for the purposes of error correction.
+
+BENCHMARKING: This licence does not prevent you from using the Deliverables
+for internal benchmarking purposes. However, you shall treat any and all
+benchmarking data, and any other results of your use or testing of the
+Deliverables which are indicative of performance, efficacy, reliability or
+quality, as confidential information and you shall not disclose such
+information to any third party without the express written permission of Arm.
+
+RESTRICTIONS ON TRANSFER OF LICENSED RIGHTS: You shall not rent or lease the
+Deliverables. Except as identified in the “PERMITTED USERS” paragraph above,
+you shall not share the Deliverables with contractors or third parties. The
+rights granted to you under this Licence may not be assigned, sublicensed or
+otherwise transferred by you to any third party without the prior written
+consent of Arm. An assignment shall be deemed to include, without limitation;
+(i) any transaction or series of transactions whereby a third party acquires,
+directly or indirectly, the power to control the management and policies of
+you, whether through the acquisition of voting securities, by contract or
+otherwise; or (ii) the sale of more than fifty percent (50%) of the your
+assets whether in a single transaction or series of transactions. You shall
+not rent or lease the Deliverables. You shall not share the Deliverables with
+contractors (except as identified in the ‘PERMITTED USERS’ clause above) or
+other third parties.
+
+COPYRIGHT AND RESERVATION OF RIGHTS: The Deliverables are owned by Arm or its
+licensors and are protected by copyright and other intellectual property laws
+and international treaties. The Deliverables are licensed not sold. You
+acquire no rights to the Deliverables other than as expressly provided by
+this Licence. You shall not remove from the Deliverables any copyright notice
+or other notice and shall ensure that any such notice is reproduced in any
+copies of the whole or any part of the Deliverables made by you or your
+permitted users.
+
+3.  SUPPORT AND MAINTENANCE.
+If you purchased the Deliverables directly from Arm, and you are not
+receiving them as an update or upgrade or as Academic Software (defined in
+Clause 2), you are entitled to reasonable support and maintenance for the
+Deliverables for the period of six (6) months from the date of purchase. The
+support will be provided on any version of the Deliverables which, at the
+date of your support request, are either; (a) the current version made
+generally available by Arm; or (b) the previous version made generally
+available by Arm at some time during the previous ninety (90) days.
+
+Support will be provided by telephone, email or other written format
+designated by Arm, prioritised at Arm’s discretion, and may not be used as a
+substitute for training or as additional resource for your programming
+projects.  Maintenance will be provided in the form of upgrades, updates and
+patch releases to the Deliverables as and when they are made generally
+available from Arm.
+
+Arm’s obligation under this Clause 3 is limited to the provision of support
+and maintenance to you and Arm is under no obligation to provide any support
+and maintenance to any third parties under this Licence. If you purchase
+support and maintenance for additional years it will be provided pursuant to
+this Clause 3 and will be subject to the terms and conditions of this Licence.
+
+If you are receiving the Deliverables as an update, you obtain no rights to,
+and shall not, install or use the update, as applicable, unless you have
+first ceased all use of, and deleted your Deliverables for the version of the
+Deliverables that you are updating or upgrading, as applicable. Future
+releases of the Deliverables might introduce backward incompatible changes.
+Please refer to product documentation for the changes in each release and for
+guidance about compatibility.
+
+If; (i) you obtained the Deliverables from an Arm authorised reseller or
+other third party; (ii) Deliverables were provided free of charge or for
+evaluation; or (iii) it is Academic Software, you are not entitled to any
+support for the Deliverables from Arm, but Arm may, at its sole discretion
+provide limited support to you. The vendor of the Deliverables may or may not
+offer support to you for the Deliverables. Please refer to the Technical
+Support area of http://www.arm.com for contact details for Arm’s support
+service and (if applicable) other authorised support channels.  Arm shall be
+under no obligation to provide support in respect of any modifications (where
+permitted) to the Deliverables.
+
+4.  CONFIDENTIALITY.
+You acknowledge that the Deliverables, any benchmarking data and related
+information mentioned in Clause 2 contain trade secrets and confidential
+material, and you agree to maintain them in confidence and apply security
+measures no less stringent than the measures which you apply to protect your
+own like information, but not less than a reasonable degree of care, to
+prevent their unauthorised disclosure and use. Subject to any restrictions
+imposed by applicable law, the period of confidentiality shall be indefinite.
+You agree that you shall not use any such information other than in normal
+use of the Deliverables under the licences granted in this Licence. You agree
+to allow Arm to (i) disclose your confidential information to subsidiaries of
+Arm subject to terms and conditions of confidentiality substantially similar
+to those set out above in this Clause 4; and (ii) inform Arm’s partner;
+Taiwan Semiconductor Manufacturing Company Ltd, that you have entered into
+this Licence with Arm.
+
+5.  LIMITED WARRANTIES.
+For the period of ninety (90) days from the date of receipt by you of the
+Deliverables, Arm warrants to you that (i) the media on which the
+Deliverables are provided shall be free from defects in materials and
+workmanship under normal use; and (ii) the Deliverables will perform
+substantially in accordance with the accompanying documentation (if any).
+Arm's total liability and your exclusive remedy for breach of these limited
+warranties shall be limited to Arm, at Arm's option; (a) replacing the
+defective Deliverables; or (b) using reasonable efforts to correct material,
+documented, reproducible defects in the Deliverables and delivering such
+corrected Deliverables to you. Any replacement Deliverables will be warranted
+for the remainder of the original warranty period or thirty (30) days,
+whichever is the longer.
+
+EXCEPT AS PROVIDED ABOVE, YOU AGREE THAT THE DELIVERABLES ARE LICENSED “AS
+IS”, AND THAT ARM EXPRESSLY DISCLAIMS ALL REPRESENTATIONS, WARRANTIES,
+CONDITIONS OR OTHER TERMS, EXPRESS, IMPLIED OR STATUTORY, INCLUDING WITHOUT
+LIMITATION THE IMPLIED WARRANTIES OF NON- INFRINGEMENT, SATISFACTORY QUALITY,
+AND FITNESS FOR A PARTICULAR PURPOSE.
+
+YOU EXPRESSLY ASSUME ALL LIABILITIES AND RISKS, FOR USE OR OPERATION OF
+SOFTWARE APPLICATIONS, INCLUDING WITHOUT LIMITATION, APPLICATIONS DESIGNED OR
+INTENDED FOR MISSION CRITICAL APPLICATIONS, SUCH AS PACEMAKERS, WEAPONARY,
+AIRCRAFT NAVIGATION, FACTORY CONTROL SYSTEMS, ETC. SHOULD THE DELIVERABLES
+PROVE DEFECTIVE, YOU ASSUME THE ENTIRE COST OF ALL NECESSARY SERVICING,
+REPAIR OR CORRECTION.
+
+6.  LIMITATION OF LIABILITY.
+TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT SHALL ARM BE
+LIABLE FOR ANY INDIRECT, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES
+(INCLUDING LOSS OF PROFITS) ARISING OUT OF THE USE OR INABILITY TO USE THE
+DELIVERABLES WHETHER BASED ON A CLAIM UNDER CONTRACT, TORT OR OTHER LEGAL
+THEORY, EVEN IF ARM WAS ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+ARM does not seek to limit or exclude liability for death or personal injury
+arising from ARM's negligence or ARM’s fraud or willful misconduct and
+because some jurisdictions do not permit the exclusion or limitation of
+liability for consequential or incidental damages the above limitation
+relating to liability for consequential damages may not apply to you.
+
+NOTWITHSTANDING ANYTHING TO THE CONTRARY CONTAINED IN THIS LICENCE, BUT
+SUBJECT TO THE PREVIOUS PARAGRAPH, THE MAXIMUM LIABILITY OF ARM TO YOU IN
+AGGREGATE FOR ALL CLAIMS MADE AGAINST ARM IN CONTRACT TORT OR OTHERWISE UNDER
+OR IN CONNECTION WITH THE SUBJECT MATTER OF THIS LICENCE SHALL NOT EXCEED THE
+GREATER OF; (I) THE TOTAL OF SUMS PAID BY YOU TO ARM (IF ANY) FOR THIS
+LICENCE; AND (II) $10 USD.  THE EXISTENCE OF MORE THAN ONE CLAIM WILL NOT
+ENLARGE OR EXTEND THE LIMIT.
+
+7.  THIRD PARTY SOFTWARE.
+The Deliverables may contain open source software, and the use of such open
+source software is expressly subject to the terms of the applicable
+license(s) for that open source software. Information about such open source
+software and applicable open source license(s) accompanies the Software.
+
+8.  GOVERNMENT END USERS.
+US Government Restrictions: Use, duplication, reproduction, release,
+modification, disclosure or transfer of the Deliverables is restricted in
+accordance with the terms of this Licence.
+
+9.  TERM AND TERMINATION.
+This Licence shall remain in force until terminated by you or by Arm. Without
+prejudice to any of its other rights if you are in breach of any of the terms
+and conditions of this Licence then Arm may terminate this Licence
+immediately upon giving written notice to you. You may terminate this Licence
+at any time. Upon termination of this Licence by you or by Arm: (i) you shall
+stop using the Deliverables and confidential information and destroy all
+copies of the Deliverables and confidential information in your possession
+together with all documentation and related materials; and (ii) Arm’s
+obligation to provide support and maintenance shall terminate immediately.
+The provisions of Clauses 4, 6, 7, 8, 9 and 10 shall survive termination of
+this Licence.
+
+10.  GENERAL.
+This Licence is governed by English law. Except where Arm agrees otherwise
+in; (i) a written contract signed by you and ARM; or (ii) a written contract
+provided by Arm and accepted by you, this is the only agreement between you
+and Arm relating to the Deliverables and it may only be modified by written
+agreement between you and Arm. This Licence may not be modified by purchase
+orders, advertising or other representation by any person. If any clause or
+sentence in this Licence is held by a court of law to be illegal or
+unenforceable the remaining provisions of this Licence shall not be affected
+thereby. The failure by Arm to enforce any of the provisions of this Licence,
+unless waived in writing, shall not constitute a waiver of Arm's rights to
+enforce such provision or any other provision of this Licence in the future.
+
+The Deliverables provided under this Licence are subject to U.S. export
+control laws, including the U.S. Export Administration Act and its associated
+regulations, and may be subject to export or import regulations in other
+countries. You agree to comply fully with all laws and regulations of the
+United States and other countries ("Export Laws") to assure that the
+Deliverables, are not (1) exported, directly or indirectly, in violation of
+Export Laws, either to any countries that are subject to U.S.A. export
+restrictions or to any end user who has been prohibited from participating in
+the U.S.A. export transactions by any federal agency of the U.S.A.
+government; or (2) intended to be used for any purpose prohibited by Export
+Laws, including, without limitation, nuclear, chemical, or biological weapons
+proliferation.
+
+ARM contract references: LES-PRE-21570

docs/n1sdp/change-log.rst

+**********
+Change Log
+**********
+
+.. contents::
+
+This document contains a summary of the incremental features, changes, fixes
+and known issues in each release of the N1SDP stack. It is listed in the order
+of latest to oldest.
+
+Tagged Version - N1SDP-2021.10.12
+=================================
+Features and Fixes
+------------------
+- Migration of source code repository from `ARM linaro GIT`_ to `ARM Gitlab`_.
+- Migration of build system from Yocto to BASH based scripting environment.
+- Minimal BusyBox based root filesystem support.
+- Rebase to the latest stable Linux kernel version ``5.10.61``.
+- Migrate to the latest stable EDK2 version ``edk2-stable202108``.
+- Update Grub to latest release tag grub-2.06.
+- Migrate to the latest master version of System Control Processor (SCP)
+  firmware and Trusted Firmware-A components.
+- MCC Refresh to v117 to include:
+
+  - QSPI programming speed improvements.
+  - Case fans can be controlled through FAN_SPEED IOFPGA register.
+
+Precautions
+-----------
+- The system thermal monitoring and control features are not yet calibrated,
+  therefore do not operate the unit above room temperature (approximately 25°C).
+- The N1SDP is intended for use within a laboratory or engineering development
+  environment. Do not use the N1SDP near equipment that is sensitive to
+  electromagnetic emissions, for example, medical equipment.
+- Never subject the board to high electrostatic potentials.
+  Observe Electrostatic Discharge (ESD) precautions when handling any board.
+
+  - Always wear a grounding strap when handling the board.
+  - Avoid touching the component pins or any other metallic elements.
+- Update/Change board firmware only if MCC FW ask to do so,
+  refer to `potential damage`_ page for more information
+- Kindly note, the USB 3.0 ports and the audio jacks available on the front
+  panel of the N1SDP case are NOT connected and are not usable. They will be
+  removed in the later versions.
+
+Known Issues and Limitations
+----------------------------
+- Patches providing proof of concept support for Xilinx CCIX accelerator
+  endpoints are no longer included in this release.
+- PCIe root port is limited to GEN3 speed due to the on-board PCIe
+  switch itself only supporting up to GEN3 speed.
+- Page Request Interface (PRI) feature is not available in both SMMUs
+  interfacing with the PCIe root ports.
+- Currently only Micron 8GB single Rank DIMMs
+  (part number: MTA9ASF1G72PZ-2G6D1) and 16GB dual Rank DIMMs
+  (part number:MTA18ASF2G72PDZ-2G6E1) are supported.
+- Stability issues have been observed on long stress tests of the
+  system.
+- On-board HDMI connection is not supported for graphics output. A PCIe
+  graphics card can be used for graphics support.
+- If either of the two boards needs to boot up in a single chip mode
+  with a C2C setup, then the other board should be powered off.
+- CCIX port on N1SDP as a PCIe root host is not supported in UEFI EDK2.
+
+Disclaimer
+----------
+- Limited Testing for now due to the current global scenario, to be revisited once
+  we get back on site.
+
+Support
+-------
+For support email: support-subsystem-enterprise@arm.com
+
+Change logs for the previous releases
+=====================================
+- Refer to the `Old Change Log`_ for detailed information on software features
+  and changes for the previous releases.
+
+.. _ARM linaro GIT: https://git.linaro.org/landing-teams/working/arm
+.. _ARM Gitlab: https://gitlab.arm.com/arm-reference-solutions
+.. _Old Change Log: https://git.linaro.org/landing-teams/working/arm/arm-reference-platforms.git/tree/change-log.rst?h=n1sdp-v2
+.. _potential damage: https://community.arm.com/developer/tools-software/oss-platforms/w/docs/604/notice-potential-damage-to-n1sdp-boards-if-using-latest-firmware-release
+
+----------
+
+*Copyright (c) 2020-2021, Arm Limited. All rights reserved.*

docs/n1sdp/cmn-performance-analysis.rst

+***************************************
+CMN-600 perf example on Neoverse N1 SDP
+***************************************
+
+.. contents::
+
+The goal of this document is to give a short introduction on CMN-600
+performance analysis on N1SDP.
+This includes driver load verification and Linux perf usage examples.
+
+The examples also include system level cache access and traffic to
+and from PCIe devices from the view of the interconnect.
+
+Support in Arm's Neoverse N1 SDP software release
+-------------------------------------------------
+The software support for CMN-600 performance analysis can be divided into three
+components:
+
+* The user space Linux perf tool
+* The Linux kernel arm-cmn driver
+* EDK2 (DSDT table entry)
+
+The default build of the supplied N1SDP software stack will include all
+necessary changes and patches to test and explore CMN-600 performance analysis.
+
+
+CMN-600 Topology and NodeIDs on Neoverse N1 SDP
+-----------------------------------------------
+The PMUs in CMN-600 are distributed to the nodes of the mesh interconnect.
+NodeType specific events are configured per node.
+Event counting is done by local counters in the XP attached to the node.
+Global counters are in the Debug Trace Controller (DTC).
+The arm-cmn driver uses local/global register pairing to provide 64-bit event
+counters (see "Counter Allocation" section below).
+
+All the nodes are referenced by NodeID and NodeType.
+PMU events must specify the NodeID of the node on which it is to be counted
+using the nodeid= parameter.
+A summary of NodeID can be found in the table below.
+For more details contact support (support-subsystem-enterprise@arm.com).
+
++---------------------------------+-----------+---------+--------------------+
+| Purpose                         | Node Type | NodeID  | Event Name         |
++---------------------------------+-----------+---------+--------------------+
+| System-Level Cache slices (SLC) | HN-F      | 0x24    | arm_cmn/hnf        |
+|                                 |           | 0x28    |                    |
+|                                 |           | 0x44    |                    |
+|                                 |           | 0x48    |                    |
++---------------------------------+-----------+---------+--------------------+
+| PCI_CCIX (Expansion slot 4)     | RN-D      | 0x08    | arm_cmn/rnid       |
++---------------------------------+-----------+---------+--------------------+
+| PCI_0 (All other PCI-E)         | RN-D      | 0x0c    | arm_cmn/rnid       |
++---------------------------------+-----------+---------+--------------------+
+| Mesh interconnections           | XP        | 0x00    | arm_cmn/mxp        |
+|                                 |           | 0x08    |                    |
+|                                 |           | 0x20    |                    |
+|                                 |           | 0x28    |                    |
+|                                 |           | 0x40    |                    |
+|                                 |           | 0x48    |                    |
+|                                 |           | 0x60    |                    |
+|                                 |           | 0x68    |                    |
++---------------------------------+-----------+---------+--------------------+
+| Debug Trace Controller          | DTC       | 0x68    | arm_cmn/dtc_cycles |
++---------------------------------+-----------+---------+--------------------+
+| ACE-lite slave                  | SBSX      | 0x64    | arm_cmn/sbsx       |
++---------------------------------+-----------+---------+--------------------+
+
+For details on what is connected to PCI_0 check the N1SDP TRM (Figure 2-9
+PCI Express and CCIX system).
+
+Software components
+-------------------
+
+Linux perf tool
+###############
+No modifications of ``perf`` source is needed.
+The user can opt to use any perf compatible with the built kernel or use the
+included script ``build-scripts/build-perf.sh`` to build a static linked binary
+from the included kernel source (binary is created as
+``output/n1sdp/perf/perf``).
+
+
+ACPI DSDT modification
+######################
+The Linux driver expects a DSDT entry that describe the location of the CMN-600
+configuration space.
+This is included in the supplied N1SDP software stack.
+
+Linux perf driver (arm-cmn)
+###########################
+The included arm-cmn driver is a work-in-progress.
+A Snapshot of this driver is included in the supplied N1SDP software stack.
+The driver is controlled by ``CONFIG_ARM_CMN`` (enabled in default software
+stack build).
+
+Counter Allocation/Limitation
+*****************************
+The arm-cmn driver provides 64-bit event counts for any given event.
+It accomplishes this using a combination of combined-pair local counters (in a
+DTM/XP) and uncombined global counters (in the DTC):
+
+* DTM/XP
+    Can provide up to two 32-bit local counters (built from paired 16-bit
+    counters por_dtm_pmevcnt0+1, and 2+3) for events from itself and/or
+    up to two devices that are connected to its ports.
+
+    Overflows from these counters are sent to its DTC's global counters.
+    This means only up to 2 events from any of the devices connected to an XP
+    can be counted at the same time without sampling.
+
+
+* DTC
+    Each DTC can provide up to 8 global counters (por_dt_pmevcntA .. H).
+    This means only up to 8 events in a DTC domain can be counted at the same
+    time without sampling.
+
+For example, the N1SDP's two PCI-Express root complexes RND (PCI_CCIX on RND3
+at NodeID 0x8 and PCI0 on RND4 at NodeID 0xC), hang off of the same XP (0,1).
+Only up to 2 RND events from either of the two PCI-E domains can be measured
+simultaneously without sampling; 3 or more will require sampling.
+
+In the following example, we try to measure 4 RND events, but perf is only
+giving 50% sampling time for each count because the events have to share local
+counters in the XP.
+::
+
+ $ perf stat -a \
+     -e arm_cmn/rnid_txdat_flits,nodeid=8/ \
+     -e arm_cmn/rnid_txdat_flits,nodeid=12/ \
+     -e arm_cmn/rnid_rxdat_flits,nodeid=8/ \
+     -e arm_cmn/rnid_rxdat_flits,nodeid=12/ \
+     -I 1000
+ #   time       counts                unit events
+ 1.000089438       0      arm_cmn/rnid_txdat_flits,nodeid=8/     (50.00%)
+ 1.000089438       0      arm_cmn/rnid_txdat_flits,nodeid=12/    (50.00%)
+ 1.000089438       0      arm_cmn/rnid_rxdat_flits,nodeid=8/     (50.00%)
+ 1.000089438       0      arm_cmn/rnid_rxdat_flits,nodeid=12/    (50.00%)
+ 2.000231897      79      arm_cmn/rnid_txdat_flits,nodeid=8/     (50.01%)
+ 2.000231897       0      arm_cmn/rnid_txdat_flits,nodeid=12/    (50.01%)
+ 2.000231897       0      arm_cmn/rnid_rxdat_flits,nodeid=8/     (49.99%)
+
+PMU Events
+**********
+``perf list`` shows the perfmon events for the node types that are detected by
+the arm-cmn driver.
+If a node type is not detected, perf list will not show the events for that
+node type.
+::
+
+    # perf list | grep arm_cmn/hnf
+    arm_cmn/hnf_brd_snoops_sent/                       [Kernel PMU event]
+    arm_cmn/hnf_cache_fill/                            [Kernel PMU event]
+    arm_cmn/hnf_cache_miss/                            [Kernel PMU event]
+    arm_cmn/hnf_cmp_adq_full/                          [Kernel PMU event]
+    arm_cmn/hnf_dir_snoops_sent/                       [Kernel PMU event]
+    arm_cmn/hnf_intv_dirty/                            [Kernel PMU event]
+    arm_cmn/hnf_ld_st_swp_adq_full/                    [Kernel PMU event]
+    arm_cmn/hnf_mc_reqs/                               [Kernel PMU event]
+    arm_cmn/hnf_mc_retries/                            [Kernel PMU event]
+    [...]
+
+The perfmon events are described in the CMN-600 TRM in the register description
+section for each node type's perf event selection register (at offset 0x2000 of
+each node that has a PMU).
+
+`CMN-600 TRM register summary`_ links to all of the node types and offset
+registers.
+
+Specifying NodeID to events in perf
+***********************************
+To program the CMN-600's PMUs, the NodeIDs of the components need to be
+specified for each event using a nodeid= parameter.
+Example:
+::
+
+ $ perf stat -a -I 1000 -e arm_cmn/hnf_mc_reqs,nodeid=0x24/
+
+Multiple nodes can be specified for an event as shown below :
+::
+
+    $ perf stat -a -I 1000 \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x24/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x28/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x44/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x48/
+
+Separate events on the same nodes can be specified as shown below :
+::
+
+    $ perf stat -a -I 1000 \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x24/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x28/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x44/ \
+        -e arm_cmn/hnf_mc_reqs,nodeid=0x48/ \
+        -e arm_cmn/hnf_mc_retries,nodeid=0x24/ \
+        -e arm_cmn/hnf_mc_retries,nodeid=0x28/ \
+        -e arm_cmn/hnf_mc_retries,nodeid=0x44/ \
+        -e arm_cmn/hnf_mc_retries,nodeid=0x48/
+
+Driver verification
+-------------------
+To verify that the arm-cmn has successfully loaded different ways:
+
+* Check if any arm_cmn entires is available
+    ::
+
+     $ perf list | grep arm_cmn
+     arm_cmn/dn_rxreq_dvmop/                            [Kernel PMU event]
+     arm_cmn/dn_rxreq_dvmop_vmid_filtered/              [Kernel PMU event]
+     arm_cmn/dn_rxreq_dvmsync/                          [Kernel PMU event]
+     arm_cmn/dn_rxreq_retried/                          [Kernel PMU event]
+     arm_cmn/dn_rxreq_trk_occupancy_all/                [Kernel PMU event]
+     arm_cmn/dn_rxreq_trk_occupancy_dvmop/              [Kernel PMU event]
+     [...]
+
+
+* Sysfs entries
+    ::
+