基础知识

进程、线程、协程

调度模型

内核级线程模型

内核级线程模型中用户线程与内核线程是一对一关系（1 : 1）。线程的创建、销毁、切换工作都是有内核完成的。应用程序不参与线程的管理工作，只能调用内核级线程编程接口(应用程序创建一个新线程或撤销一个已有线程时，都会进行一个系统调用）。每个用户线程都会被绑定到一个内核线程。用户线程在其生命期内都会绑定到该内核线程。一旦用户线程终止，两个线程都将离开系统。

操作系统调度器管理、调度并分派这些线程。运行时库为每个用户级线程请求一个内核级线程。操作系统的内存管理和调度子系统必须要考虑到数量巨大的用户级线程。操作系统为每个线程创建上下文。进程的每个线程在资源可用时都可以被指派到处理器内核。

内核级线程模型有如下优点：

缺点：

用户级线程模型

用户线程模型中的用户线程与内核线程是多对一关系（N : 1）。线程的创建、销毁以及线程之间的协调、同步等工作都是在用户态完成，具体来说就是由应用程序的线程库控制。内核对用户态的调度是无感知的，内核此时的调度都是基于主线程的。线程的并发处理从宏观来看，任意时刻每个进程只能够有一个线程在运行，且只有一个 CPU 核心会被分配给该进程。

用户级线程有如下优点：

缺点：

协程模型

协程模型中用户协程与协程调度器绑定，协程调度器在绑定内核线程后执行代码逻辑。协程模型充分吸收上面两种模型的优点，尽量规避缺点。协程创建在用户空间中完成，线程的调度和同步也在应用程序中进行。一个应用程序中的多个协程被绑定到一些（小于或等于用户级线程的数目）内核级线程上。

Plan9 汇编

常用寄存器

SP (Stack Pointer)

SP 寄存器是栈指针寄存器，它总是指向栈顶的当前位置。在函数调用和返回时，以及在局部变量分配时，SP 寄存器会被频繁使用。在 Go 的 Plan 9 汇编中，它通常表示为 SP 或者 RSP（在 64 位 x86 架构上）。

SB (Static Base)

SB 在 Go 的 Plan 9 汇编中是一个伪寄存器，代表静态基址。它用于表示全局地址空间的起点，是全局变量和函数地址的引用基点。SB 不对应任何实际的硬件寄存器。

AX (Accumulator)

AX 寄存器是累加器寄存器，通常用于算术运算、数据传输、I/O 操作和一些特定的指令。在 64 位 x86 架构中，它被扩展为 RAX。在函数调用时，RAX 通常用于存储函数的返回值。

BX (Base)

BX 寄存器是基址寄存器，它可以用于作为地址计算的基点。在 64 位 x86 架构中，BX 被扩展为 RBX。RBX 是少数几个在函数调用时需要被调用者保存（callee-saved）的寄存器之一，这意味着如果一个函数要使用 RBX，它必须在使用前将原来的值压栈，使用后再恢复。

CX (Count)

CX 寄存器是计数寄存器，它经常用于循环和字符串操作指令中表示计数。在 64 位 x86 架构中，它被扩展为 RCX。在某些操作中，如 REP 前缀的字符串操作指令，RCX 寄存器用来计数重复操作的次数。

DX(Data)

DX 寄存器是数据寄存器，用于 I/O 操作和一些算术运算。在 64 位 x86 架构中，它被扩展为 RDX。

PC (Program Counter)

PC 被用来指代当前的指令地址。但是，它的使用方式可能与传统汇编有所不同。在 Go 汇编中，PC 更多地作为一个抽象的概念出现，在不同的架构上会被映射到指定架构的 PC 寄存器。

BP (Base Pointer)

BP寄存器，是表示已给调用栈的起始栈底（栈的方向从大到小，SP表示栈顶），仅部分架构支持，保存栈基地址能较方便地进行栈展开和栈分裂。

GMP 模型

概念定义

每个 goroutine (G) 都在操作系统线程 (M) 上运行，该线程分配给一个逻辑处理器 (P)。

调度模型

GMP 内部结构

G

// 完整结构参考：https://github.com/golang/go/blob/go1.22.1/src/runtime/runtime2.go

type g struct {
  // Stack parameters.
  // stack describes the actual stack memory: [stack.lo, stack.hi).
  // stackguard0 is the stack pointer compared in the Go stack growth prologue.
  // It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption.
  // stackguard1 is the stack pointer compared in the //go:systemstack stack growth prologue.
  // It is stack.lo+StackGuard on g0 and gsignal stacks.
  // It is ~0 on other goroutine stacks, to trigger a call to morestackc (and crash).
  stack       stack   // offset known to runtime/cgo
  stackguard0 uintptr // offset known to liblink
  stackguard1 uintptr // offset known to liblink

  _panic    *_panic // innermost panic - offset known to liblink
  _defer    *_defer // innermost defer
  m         *m      // current m; offset known to arm liblink
  sched     gobuf
  syscallsp uintptr // if status==Gsyscall, syscallsp = sched.sp to use during gc
  syscallpc uintptr // if status==Gsyscall, syscallpc = sched.pc to use during gc
  stktopsp  uintptr // expected sp at top of stack, to check in traceback
  // param is a generic pointer parameter field used to pass
  // values in particular contexts where other storage for the
  // parameter would be difficult to find. It is currently used
  // in four ways:
  // 1. When a channel operation wakes up a blocked goroutine, it sets param to
  //    point to the sudog of the completed blocking operation.
  // 2. By gcAssistAlloc1 to signal back to its caller that the goroutine completed
  //    the GC cycle. It is unsafe to do so in any other way, because the goroutine's
  //    stack may have moved in the meantime.
  // 3. By debugCallWrap to pass parameters to a new goroutine because allocating a
  //    closure in the runtime is forbidden.
  // 4. When a panic is recovered and control returns to the respective frame,
  //    param may point to a savedOpenDeferState.
  param        unsafe.Pointer
  atomicstatus atomic.Uint32
  stackLock    uint32 // sigprof/scang lock; TODO: fold in to atomicstatus
  goid         uint64
  schedlink    guintptr
  waitsince    int64      // approx time when the g become blocked
  waitreason   waitReason // if status==Gwaiting

  preempt       bool // preemption signal, duplicates stackguard0 = stackpreempt
  preemptStop   bool // transition to _Gpreempted on preemption; otherwise, just deschedule
  preemptShrink bool // shrink stack at synchronous safe point
  ...
  lockedm       muintptr
  sig           uint32
  writebuf      []byte
  sigcode0      uintptr
  sigcode1      uintptr
  sigpc         uintptr
  parentGoid    uint64          // goid of goroutine that created this goroutine
  gopc          uintptr         // pc of go statement that created this goroutine
  ancestors     *[]ancestorInfo // ancestor information goroutine(s) that created this goroutine (only used if debug.tracebackancestors)
  startpc       uintptr         // pc of goroutine function
  racectx       uintptr
  waiting       *sudog         // sudog structures this g is waiting on (that have a valid elem ptr); in lock order
  ...
  selectDone    atomic.Uint32  // are we participating in a select and did someone win the race?
  ...
  // Per-G GC state

  // gcAssistBytes is this G's GC assist credit in terms of
  // bytes allocated. If this is positive, then the G has credit
  // to allocate gcAssistBytes bytes without assisting. If this
  // is negative, then the G must correct this by performing
  // scan work. We track this in bytes to make it fast to update
  // and check for debt in the malloc hot path. The assist ratio
  // determines how this corresponds to scan work debt.
  gcAssistBytes int64
}

执行栈管理相关字段

语言特性相关字段

执行环境和调度相关字段

Goroutine 状态和控制字段

抢占相关字段

GC 相关字段

信号处理相关字段

其他字段

G 的状态

Presudo-G

// 完整结构参考：https://github.com/golang/go/blob/go1.22.1/src/runtime/runtime2.go

type sudog struct {
  // The following fields are protected by the hchan.lock of the
  // channel this sudog is blocking on. shrinkstack depends on
  // this for sudogs involved in channel ops.

  g *g

  next *sudog
  prev *sudog
  elem unsafe.Pointer // data element (may point to stack)

  // The following fields are never accessed concurrently.
  // For channels, waitlink is only accessed by g.
  // For semaphores, all fields (including the ones above)
  // are only accessed when holding a semaRoot lock.

  acquiretime int64
  releasetime int64
  ticket      uint32

  // isSelect indicates g is participating in a select, so
  // g.selectDone must be CAS'd to win the wake-up race.
  isSelect bool

  // success indicates whether communication over channel c
  // succeeded. It is true if the goroutine was awoken because a
  // value was delivered over channel c, and false if awoken
  // because c was closed.
  success bool

  // waiters is a count of semaRoot waiting list other than head of list,
  // clamped to a uint16 to fit in unused space.
  // Only meaningful at the head of the list.
  // (If we wanted to be overly clever, we could store a high 16 bits
  // in the second entry in the list.)
  waiters uint16

  parent   *sudog // semaRoot binary tree
  waitlink *sudog // g.waiting list or semaRoot
  waittail *sudog // semaRoot
  c        *hchan // channel
}

Presudo-G 是处于等待列表的 G 的替身，例如等待往 Channel 发送或从 Channel 接收信息的 Goroutine。之所以要设计替身结构是因为单个 Goroutine 可能同时身处对个等待列表上，因此 Goroutine 和同步对象之间是多对多的关系。

M

// 完整结构参考：https://github.com/golang/go/blob/go1.22.1/src/runtime/runtime2.go

type m struct {
  g0      *g     // goroutine with scheduling stack
  morebuf gobuf  // gobuf arg to morestack
  divmod  uint32 // div/mod denominator for arm - known to liblink
  _       uint32 // align next field to 8 bytes

  // Fields not known to debuggers.
  procid        uint64            // for debuggers, but offset not hard-coded
  gsignal       *g                // signal-handling g
  goSigStack    gsignalStack      // Go-allocated signal handling stack
  sigmask       sigset            // storage for saved signal mask
  tls           [tlsSlots]uintptr // thread-local storage (for x86 extern register)
  mstartfn      func()
  curg          *g       // current running goroutine
  caughtsig     guintptr // goroutine running during fatal signal
  p             puintptr // attached p for executing go code (nil if not executing go code)
  nextp         puintptr
  oldp          puintptr // the p that was attached before executing a syscall
  id            int64
  ...
  syscalltick uint32
  freelink    *m // on sched.freem
  trace       mTraceState
}

调度相关字段

P

// 完整结构参考：https://github.com/golang/go/blob/go1.22.1/src/runtime/runtime2.go

type p struct {
  id          int32
  status      uint32 // one of pidle/prunning/...
  link        puintptr
  schedtick   uint32     // incremented on every scheduler call
  syscalltick uint32     // incremented on every system call
  sysmontick  sysmontick // last tick observed by sysmon
  m           muintptr   // back-link to associated m (nil if idle)
  mcache      *mcache
  pcache      pageCache
  raceprocctx uintptr

  deferpool    []*_defer // pool of available defer structs (see panic.go)
  deferpoolbuf [32]*_defer

  // Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen.
  goidcache    uint64
  goidcacheend uint64

  // Queue of runnable goroutines. Accessed without lock.
  runqhead uint32
  runqtail uint32
  runq     [256]guintptr
  // runnext, if non-nil, is a runnable G that was ready'd by
  // the current G and should be run next instead of what's in
  // runq if there's time remaining in the running G's time
  // slice. It will inherit the time left in the current time
  // slice. If a set of goroutines is locked in a
  // communicate-and-wait pattern, this schedules that set as a
  // unit and eliminates the (potentially large) scheduling
  // latency that otherwise arises from adding the ready'd
  // goroutines to the end of the run queue.
  //
  // Note that while other P's may atomically CAS this to zero,
  // only the owner P can CAS it to a valid G.
  runnext guintptr

  // Available G's (status == Gdead)
  gFree struct {
   gList
   n int32
  }

  sudogcache []*sudog
  sudogbuf   [128]*sudog
}

调度相关字段

P 的状态

Stack

stack 结构体主要用来记录 goroutine 所使用的栈的信息，包括栈顶和栈底位置。

// Stack describes a Go execution stack.
// The bounds of the stack are exactly [lo, hi),
// with no implicit data structures on either side.
// 用于记录goroutine使用的栈的起始和结束位置
type stack struct{  
    lo uintptr   // 栈顶，指向内存低地址
    hi uintptr   // 栈底，指向内存高地址
}

Gobuf

gobuf 结构体用于保存 goroutine 的调度信息。

type gobuf struct {
  // The offsets of sp, pc, and g are known to (hard-coded in) libmach.
  //
  // ctxt is unusual with respect to GC: it may be a
  // heap-allocated funcval, so GC needs to track it, but it
  // needs to be set and cleared from assembly, where it's
  // difficult to have write barriers. However, ctxt is really a
  // saved, live register, and we only ever exchange it between
  // the real register and the gobuf. Hence, we treat it as a
  // root during stack scanning, which means assembly that saves
  // and restores it doesn't need write barriers. It's still
  // typed as a pointer so that any other writes from Go get
  // write barriers.
  sp   uintptr  // 栈指针，保存了 g 的栈顶地址。在 g 被暂停时，sp 会被设置为当前的栈顶。
  pc   uintptr  // 程序计数器，保存了下一条要执行的指令的地址。当 g 被恢复执行时，会从这个地址开始执行。
  g    guintptr // 当前 gobuf 对象所属 goroutine 指针
  
  // 上下文指针，用于保存函数调用时的额外上下文信息。
  // 在垃圾回收时，它需要特殊处理，因为它可能指向堆上的对象。
  // 由于在汇编代码中处理写屏障（write barriers）比较困难，ctxt 被当作根（root）来处理。
  // 这意味着它在栈扫描时是活跃的，并且汇编代码中保存和恢复它的操作不需要写屏障。
  ctxt unsafe.Pointer
  
  
  ret  uintptr // 保存函数调用的返回值
  lr   uintptr // 链接寄存器，在那些使用链接寄存器的架构中（如 ARM），保存了函数调用返回的地址
  
  bp   uintptr // 基指针，在启用了帧指针的架构中保存当前栈帧的基地址。这对于调试和堆栈展开非常有用
}

调度器

调度器内部结构

// 完整结构参考：https://github.com/golang/go/blob/go1.22.1/src/runtime/runtime2.go

type schedt struct {
    lock mutex
    
    // When increasing nmidle, nmidlelocked, nmsys, or nmfreed, be
    // sure to call checkdead().
    
    midle        muintptr // 空闲的 m
    nmidle       int32    // 空闲的 m 数量
    nmidlelocked int32    // number of locked m's waiting for work
    mnext        int64    // number of m's that have been created and next M ID
    maxmcount    int32    // maximum number of m's allowed (or die)
    nmsys        int32    // number of system m's not counted for deadlock
    nmfreed      int64    // cumulative number of freed m's
    
    ngsys atomic.Int32 // number of system goroutines
    
   
    pidle        puintptr      // 空闲的 P
    npidle       atomic.Int32  // 空闲 P 数量
    nmspinning   atomic.Int32  // See "Worker thread parking/unparking" comment in proc.go.
    needspinning atomic.Uint32 // See "Delicate dance" comment in proc.go. Boolean. Must hold sched.lock to set to 1.
    
    // Global runnable queue.
    // 全局 _GRunnable G 队列
    runq     gQueue
    runqsize int32
    
    // disable controls selective disabling of the scheduler.
    //
    // Use schedEnableUser to control this.
    //
    // disable is protected by sched.lock.
    disable struct {
      // user disables scheduling of user goroutines.
      user     bool
      runnable gQueue // pending runnable Gs
      n        int32  // length of runnable
    }
    
    // Global cache of dead G's.
    // 全局的 GFree 缓存队列，可供 P 获取
    gFree struct {
      lock    mutex
      stack   gList // Gs with stacks
      noStack gList // Gs without stacks
      n       int32
    }
    
    // Central cache of sudog structs.
    sudoglock  mutex
    sudogcache *sudog
    
    // Central pool of available defer structs.
    deferlock mutex
    deferpool *_defer
    
    // freem is the list of m's waiting to be freed when their
    // m.exited is set. Linked through m.freelink.
    freem *m
}

调度器生命周期

程序首次初始化

g0，m0，工作线程

直接在栈上初始化 g0 栈空间，绑定 m0 和工作线程，并绑定 m0 和 g0。

TEXT runtime·rt0_go(SB),NOSPLIT|NOFRAME|TOPFRAME,$0
    ...
    // create istack out of the given (operating system) stack.
    // _cgo_init may update stackguard.
    // _cgo_init CGO 相关的初始化可能会更新 Stackguard，暂时忽略
    MOVQ  $runtime·g0(SB), DI        // 保存 runtime.g0 的地址到 DI
    LEAQ  (-64*1024)(SP), BX         // BX 指向线程栈 SP-64*1024+104 处，用于作为 g0 调用栈
    MOVQ  BX, g_stackguard0(DI)      // 修改 runtime.g0.stackguard0 指向 g0 栈底
    MOVQ  BX, g_stackguard1(DI)      // 修改 runtime.g0.stackguard1 指向 g0 栈底
    MOVQ  BX, (g_stack+stack_lo)(DI) // 修改 runtime.g0.satck.stack_lo 指向 g0 栈底
    MOVQ  SP, (g_stack+stack_hi)(DI) // 修改 runtime.g0.satck.stack_hi 指向 g0 栈顶
    ...
    
    LEAQ runtime·m0+m_tls(SB), DI // DI = &m0.tls，取 m0 的 tls 成员的地址到 DI 寄存器
    // 调用 settls 设置线程本地存储，settls 函数的参数在 DI 寄存器中
    // 该函数调用结果取决于不同目标操作系统对 TLS 的管理机制
    CALL runtime·settls(SB)      

    // 获取 FS 段基地址并放入 BX 寄存器，
    // 其实就是 m0.tls[1] 的地址，get_tls 的根据系统架构决定
    get_tls(BX)   
    
    // 把整型常量 0x123 拷贝到 FS 段基地址偏移 -8 的内存位置，
    // 也就是m0.tls[0] = 0x123
    MOVQ $0x123, g(BX)
    
    // AX=m0.tls[0]
    MOVQ runtime·m0+m_tls(SB), AX
    
    // 检查 m0.tls[0] 的值是否是通过线程本地存储存入的 0x123 来验证 tls 功能是否正常
    CMPQ AX, $0x123
    JEQ 2(PC)
    CALL runtime·abort(SB) // 如果线程本地存储不能正常工作，退出程序
ok:
    // set the per-goroutine and per-mach "registers"
    get_tls(BX)
    LEAQ runtime·g0(SB), CX  // CX = &g0
    MOVQ CX, g(BX)           // m0.tls[0] = &g0
    LEAQ runtime·m0(SB), AX  // AX = &m0

    // save m->g0 = g0
    MOVQ CX, m_g0(AX)
    // save m0 to g0->m
    MOVQ AX, g_m(CX)

    CLD     // convention is D is always left cleared

调度初始化

// runtime/asm_amd64.s

TEXT runtime·rt0_go(SB),NOSPLIT|NOFRAME|TOPFRAME,$0
    ... 
ok:    
    ...
    MOVL    24(SP), AX    // copy argc
    MOVL    AX, 0(SP)
    MOVQ    32(SP), AX    // copy argv
    MOVQ    AX, 8(SP)
    CALL    runtime·args(SB)       // 读取程序参数
    CALL    runtime·osinit(SB)     // 针对操作系统初始化
    CALL    runtime·schedinit(SB)  // 调度初始化
    ...

主调度初始化函数，通过信号量 STW，随后进行所有的必要的初始化操作

// runtime/proc.go
func schedinit() {
    // 初始化调度器的各种锁，防止调度过程中的并发修改
    lockInit(&sched.lock, lockRankSched)
    ...
    lockInit(&memstats.heapStats.noPLock, lockRankLeafRank)

    
    gp := getg() // gp = g0
    ...
    // 默认设置最多启动 10000 个操作系统线程，也是最多 10000 个 M
    // 决定处于执行状态的 M 的数量取决于 GOMAXPROCS 也就是 P 的数量
    sched.maxmcount = 10000 
    // 包括初始化堆栈内存池，根据 CPU 架构确定可用扩展指令集等
    ...
    mcommoninit(gp.m, -1) // 初始化 m0, gp.m = m0
    ...
    // 初始化 Ps
    procs := ncpu
    if n, ok := atoi32(gogetenv("GOMAXPROCS")); ok && n > 0 {
      procs = n
    }
    if procresize(procs) != nil {
      throw("unknown runnable goroutine during bootstrap")
    }
    ...
}

mcommoninit() 函数会对 m0 进行初始化

// Pre-allocated ID may be passed as 'id', or omitted by passing -1.
func mcommoninit(mp *m, id int64) {
  gp := getg() // gp = g0

  // g0 stack won't make sense for user (and is not necessary unwindable).
  if gp != gp.m.g0 {
     callers(1, mp.createstack[:])
  }

  lock(&sched.lock)

  // 获取线程 id
  // 并检查已创建数量是否已经超过最大值
  if id >= 0 {
     mp.id = id
  } else {
     mp.id = mReserveID()
  }

  // 初始化随机数生成相关状态
  mrandinit(mp)

  // 初始化信号处理 goroutine
  mpreinit(mp)
  if mp.gsignal != nil {
     mp.gsignal.stackguard1 = mp.gsignal.stack.lo + stackGuard
  }

  // 把新创建的 m 加入到 allm 链表上
  // Add to allm so garbage collector doesn't free g->m
  // when it is just in a register or thread-local storage.
  mp.alllink = allm

  // NumCgoCall() and others iterate over allm w/o schedlock,
  // so we need to publish it safely.
  atomicstorep(unsafe.Pointer(&allm), unsafe.Pointer(mp))
  unlock(&sched.lock)

  // Allocate memory to hold a cgo traceback if the cgo call crashes.
  if iscgo || GOOS == "solaris" || GOOS == "illumos" || GOOS == "windows" {
     mp.cgoCallers = new(cgoCallers)
  }
}

procresize() 函数对 P 进行初始化

func procresize(nprocs int32) *p {
    // 检查锁和 STW 状态
    assertLockHeld(&sched.lock)
    assertWorldStopped()
    ...
    // 首次初始化 gomaxprocs = 0
    old := gomaxprocs
    ...
    
    if nprocs > int32(len(allp)) {
        lock(&allpLock)
        // 非初始化缩容操作
        if nprocs <= int32(cap(allp)) {
            allp = allp[:nprocs]
        } else {
            // 初始化 allp Slice
            nallp := make([]*p, nprocs)
            // 初始化时 allp 为空，会跳过 copy 操作
            // 运行时扩容会将 cap 个 P 复制到新创建的 allp 中避免丢失
            copy(nallp, allp[:cap(allp)])
            allp = nallp
        }
        ...
        unlock(&allpLock)
    }
    
    // 初始化新创建的 Ps
    for i := old; i < nprocs; i++ {
        pp := allp[i]
        if pp == nil {
            pp = new(p)
        }
        // 设置 P 的 id, 状态，内存缓存等
        // 此时 P 的状态为 _Pgcstop
        pp.init(i)
        atomicstorep(unsafe.Pointer(&allp[i]), unsafe.Pointer(pp))
    }
    
    
    gp := getg() // gp = g0
    if gp.m.p != 0 && gp.m.p.ptr().id < nprocs {
        // 执行 Resize 操作的 P 不在释放范围
        // 则使用当前 P 继续下列操作
        gp.m.p.ptr().status = _Prunning
        gp.m.p.ptr().mcache.prepareForSweep()
    } else {
        // 执行 Resize 操作的 P 在释放范围
        // 则释放当前的 P,从刚初始化好的 allp 切片中获取 0 号位的 P 继续下列操作
        // 由于写屏障的限制，释放 P 的操作也要由新 P 代劳
        if gp.m.p != 0 {
            // 一些 Trace 操作
            ...
            gp.m.p.ptr().m = 0
        }
        gp.m.p = 0
        pp := allp[0]
        pp.m = 0
        pp.status = _Pidle
        acquirep(pp) // 新 P 绑定原有 P 的 M，通过 getg() 获取 g.m
        // 一些 Trace 操作
        ...
    }
    
    // 释放未使用的 P 的资源
    // 初始化时因为 old = 0 所以会跳过
    for i := nprocs; i < old; i++ {
      pp := allp[i]
      pp.destroy()
      // can't free P itself because it can be referenced by an M in syscall
    }
    ...
    var runnablePs *p
    
    // 下面这个for 循环把所有空闲的p放入空闲链表
    for i := nprocs - 1; i >= 0; i-- {
        pp := allp[i]
        // 跳过当前正在执行 resize 操作的 P
        if gp.m.p.ptr() == pp {
            continue
        }
        pp.status = _Pidle
        if runqempty(pp) {
            pidleput(pp, now)
        } else {
            // 从 schedt.midle 获取空闲的 m，可能为空
            pp.m.set(mget())
            pp.link.set(runnablePs)
            runnablePs = pp
        }
    }
    ...
    return runnablePs
}

至此，完成了 g0, m0 和 m0, p 的绑定，程序可以开始进行调度。

创建入口 Goroutine

// runtime/asm_amd64.s

// 全局变量 runtime·mainPC
// 后面 runtime·rt0_go 会使用这个入口点初始化一个 goroutine 启动程序
DATA  runtime·mainPC+0(SB)/8,$runtime·main<ABIInternal>(SB)
GLOBL runtime·mainPC(SB),RODATA,$8

TEXT runtime·rt0_go(SB),NOSPLIT|NOFRAME|TOPFRAME,$0
    ... 
ok:    
    ...
    // 创建一个新的 goroutine 加入队列以启动程序
    // 程序入口点，全局变量指向 runtime·main 函数
    MOVQ    $runtime·mainPC(SB), AX       
    PUSHQ   AX
    CALL   runtime·newproc(SB)  // 创建新 goroutine 加入执行队列 
    POPQ    AX

初始化工作都完成后，将 mainPC 指向的 runtime.main 函数传入 runtime.newproc，创建了一个新的 goroutine。

// Create a new g running fn.
// Put it on the queue of g's waiting to run.
// The compiler turns a go statement into a call to this.
func newproc(fn *funcval) {
    // 初始化时，gp = runtime.g0
    // 否则为调用方 g
    gp := getg() 
    
    // getcallerpc() 返回一个地址，
    // 也就是调用 newproc 时调用语句所在的地址
    // 用于完成 newproc 调用后继续执行原有逻辑
    pc := getcallerpc()
    
    // systemstack() 函数的作用是切换到 g0 栈执行作为参数的函数
    // 如果本身为 m.g0 或 m.gsignal 调用,则无需进行栈切换
    systemstack(func() {
        // newproc1 会初始化对应调用的 g, 绑定 m, p 等
        newg := newproc1(fn, gp, pc)

        pp := getg().m.p.ptr()
        
        // runqput 将 g 入队调度
        // 如果开启了 race detector, 会根据随机数将 next 参数置为 false
        // next 参数为 true 替换当前 P 的 runnext
        // P 原有 runnext 不为空则放入 P 本地队列
        // P 本地队列满则放入全局队列
        runqput(pp, newg, true)

        if mainStarted {
             wakep()
        }
    })
}
// Create a new g in state _Grunnable, starting at fn. callerpc is the
// address of the go statement that created this. The caller is responsible
// for adding the new g to the scheduler.
func newproc1(fn *funcval, callergp *g, callerpc uintptr) *g {
    if fn == nil {
        fatal("go of nil func value")
    }
    // 获取当前 getg().m
    // 会加锁避免争抢
    mp := acquirem() // disable preemption because we hold M and P in local vars.
    pp := mp.p.ptr()
    
    // 1. 从本地队列 p.gFree 获取空闲的 g
    // 2. 从全局队列按 schedt.gFree.stack 先，schedt.gFree.nostack 后的顺序获取空闲的 g
    // 3. 队列为空，初始化新的 g
    newg := gfget(pp)
    if newg == nil {
        newg = malg(stackMin)
        casgstatus(newg, _Gidle, _Gdead)
        allgadd(newg) // publishes with a g->status of Gdead so GC scanner doesn't look at uninitialized stack.
    }
    
    // 检查 g 栈分配
    if newg.stack.hi == 0 {
        throw("newproc1: newg missing stack")
    }
    // 检查 g 的状态
    if readgstatus(newg) != _Gdead {
        throw("newproc1: new g is not Gdead")
    }
    ...
    
    // 清空 newg.sched
    memclrNoHeapPointers(unsafe.Pointer(&newg.sched), unsafe.Sizeof(newg.sched))
    // 保存栈顶指针
    newg.sched.sp = sp
    newg.stktopsp = sp
    // newg.sched.pc表示当newg被调度起来运行时从这个地址开始执行指令
    // 当前把pc设置成了 goexit 这个函数地址 + 1（sys.PCQuantum等于1）的位置
    newg.sched.pc = abi.FuncPCABI0(goexit) + sys.PCQuantum // +PCQuantum so that previous instruction is in same function
    newg.sched.g = guintptr(unsafe.Pointer(newg))
    // 调整 gobuf(newg.sched) 和 newg 的栈空间
    gostartcallfn(&newg.sched, fn)
    newg.parentGoid = callergp.goid
    newg.gopc = callerpc
    newg.ancestors = saveAncestors(callergp)
    newg.startpc = fn.fn
    ...
    // 切换为 _Grunnable 状态
    casgstatus(newg, _Gdead, _Grunnable)
    ...
    releasem(mp)
    return newg
 }   
// adjust Gobuf as if it executed a call to fn
// and then stopped before the first instruction in fn.
func gostartcallfn(gobuf *gobuf, fv *funcval) {
  var fn unsafe.Pointer
  if fv != nil {
   fn = unsafe.Pointer(fv.fn)
  } else {
   fn = unsafe.Pointer(abi.FuncPCABIInternal(nilfunc))
  }
  gostartcall(gobuf, fn, unsafe.Pointer(fv))
}

// adjust Gobuf as if it executed a call to fn with context ctxt
// and then stopped before the first instruction in fn.
func gostartcall(buf *gobuf, fn, ctxt unsafe.Pointer) {
  sp := buf.sp
  // 从 buf.sp（栈指针）中减去指针大小（goarch.PtrSize）
  // 为即将“伪造”的函数调用腾出空间。
  // 这类似在 x86 架构上的 "CALL" 指令行为，它会将返回地址压栈。
  sp -= goarch.PtrSize
  // 将 buf.pc（程序计数器，即将要返回的地址）保存到新的栈顶位置。
  // 这模拟了函数调用时的行为——当前的 pc 值是调用函数后应该返回到的点。
  // 使得 fn 执行完后返回到 goexit 继续执行，从而完成清理工作
  *(*uintptr)(unsafe.Pointer(sp)) = buf.pc
  // 重新设置 newg 栈顶
  buf.sp = sp
  // 将真正需要执行的函数赋值给 newg.sched.pc
  // 当 goroutine 恢复执行时，它将从这个地址开始执行。
  buf.pc = uintptr(fn)
  buf.ctxt = ctxt
}

启动 m0

// runtime/asm_amd64.s

// m0 启动函数，大部分架构都指向 runtime·mstart0
TEXT runtime·mstart(SB),NOSPLIT|TOPFRAME|NOFRAME,$0
  CALL runtime·mstart0(SB)
  RET // not reached

TEXT runtime·rt0_go(SB),NOSPLIT|NOFRAME|TOPFRAME,$0
    ... 
ok:    
    ...
    // start this M
    CALL    runtime·mstart(SB)  // 启动 m0
    
    CALL    runtime·abort(SB)   // mstart should never return
    RET
    
    ... // 针对不符合要求的处理器架构的操作，报错退出
    
// mstart0 is the Go entry-point for new Ms.
// This must not split the stack because we may not even have stack
// bounds set up yet.
//
// May run during STW (because it doesn't have a P yet), so write
// barriers are not allowed.
//
//go:nosplit
//go:nowritebarrierrec
func mstart0() {
  gp := getg()

  osStack := gp.stack.lo == 0
  if osStack {
   // Initialize stack bounds from system stack.
   // Cgo may have left stack size in stack.hi.
   // minit may update the stack bounds.
   //
   // Note: these bounds may not be very accurate.
   // We set hi to &size, but there are things above
   // it. The 1024 is supposed to compensate this,
   // but is somewhat arbitrary.
   size := gp.stack.hi
   if size == 0 {
    size = 16384 * sys.StackGuardMultiplier
   }
   gp.stack.hi = uintptr(noescape(unsafe.Pointer(&size)))
   gp.stack.lo = gp.stack.hi - size + 1024
  }
  // Initialize stack guard so that we can start calling regular
  // Go code.
  gp.stackguard0 = gp.stack.lo + stackGuard
  // This is the g0, so we can also call go:systemstack
  // functions, which check stackguard1.
  gp.stackguard1 = gp.stackguard0
  mstart1()

  // Exit this thread.
  if mStackIsSystemAllocated() {
   // Windows, Solaris, illumos, Darwin, AIX and Plan 9 always system-allocate
   // the stack, but put it in gp.stack before mstart,
   // so the logic above hasn't set osStack yet.
   osStack = true
  }
  mexit(osStack)
}

// The go:noinline is to guarantee the getcallerpc/getcallersp below are safe,
// so that we can set up g0.sched to return to the call of mstart1 above.
//
//go:noinline
func mstart1() {
  gp := getg()

  // 限制 mstart 必须在 m.g0 上调用
  if gp != gp.m.g0 {
   throw("bad runtime·mstart")
  }

  // Set up m.g0.sched as a label returning to just
  // after the mstart1 call in mstart0 above, for use by goexit0 and mcall.
  // We're never coming back to mstart1 after we call schedule,
  // so other calls can reuse the current frame.
  // And goexit0 does a gogo that needs to return from mstart1
  // and let mstart0 exit the thread.
  gp.sched.g = guintptr(unsafe.Pointer(gp))
  // 保存再次运行时的指令地址，即返回后会执行线程退出操作
  gp.sched.pc = getcallerpc()
  // 保存再次运行时的栈顶
  gp.sched.sp = getcallersp()

  // 部分 CPU 架构要进行特殊的初始化
  asminit()
  // 信号相关初始化
  minit()

  // Install signal handlers; after minit so that minit can
  // prepare the thread to be able to handle the signals.
  // 针对 M0 的信号初始化
  if gp.m == &m0 {
   mstartm0()
  }

  // 执行启动函数
  // 首次初始化 g0.m.mstart = nil
  if fn := gp.m.mstartfn; fn != nil {
   fn()
  }

  // m0已经绑定了 allp[0]，不是 m0 的话还没有 p，所以需要获取一个 p
  if gp.m != &m0 {
   acquirep(gp.m.nextp.ptr())
   gp.m.nextp = 0
  }
  
  // 开始调度
  schedule()
}

执行调度

// One round of scheduler: find a runnable goroutine and execute it.
// Never returns.
func schedule() {
  // 初始化时是 m0.g0, 其他为 m.g0
  mp := getg().m
  ...
top:
  ...
  gp, inheritTime, tryWakeP := findRunnable() // 阻塞获取 Runnable goroutine
  ...
  execute(gp, inheritTime) // 执行获取到的 goroutine
}
// Finds a runnable goroutine to execute.
// Tries to steal from other P's, get g from local or global queue, poll network.
// tryWakeP indicates that the returned goroutine is not normal (GC worker, trace
// reader) so the caller should try to wake a P.
func findRunnable() (gp *g, inheritTime, tryWakeP bool) {
  mp := getg().m
  ...
  // Try to schedule a GC worker.
  // GC 辅助任务
  if gcBlackenEnabled != 0 {
    gp, tnow := gcController.findRunnableGCWorker(pp, now)
    if gp != nil {
      return gp, false, true
    }
    now = tnow
  }
  
  // 为了保证调度的公平性，每进行61次调度就需要优先从全局运行队列中获取goroutine，
  // 因为如果只调度本地队列中的g，那么全局运行队列中的goroutine将得不到运行
  if pp.schedtick%61 == 0 && sched.runqsize > 0 {
    lock(&sched.lock)
    gp := globrunqget(pp, 1)
    unlock(&sched.lock)
    if gp != nil {
      return gp, false, false
    }
  }
}
// Schedules gp to run on the current M.
// If inheritTime is true, gp inherits the remaining time in the
// current time slice. Otherwise, it starts a new time slice.
// Never returns.
//
// Write barriers are allowed because this is called immediately after
// acquiring a P in several places.
//
//go:yeswritebarrierrec
func execute(gp *g, inheritTime bool) {
  // 获取当前 m
  // 初始化时是 m0
  mp := getg().m
  ...
  // 切换 curg 为待调度的 newg
  mp.curg = gp
  gp.m = mp
  
  // 切换状态
  casgstatus(gp, _Grunnable, _Grunning)
  gp.waitsince = 0
  gp.preempt = false
  gp.stackguard0 = gp.stack.lo + stackGuard
  if !inheritTime {
   mp.p.ptr().schedtick++
  }

  ...
  // 执行  
  gogo(&gp.sched)
}
// func gogo(buf *gobuf)
// restore state from Gobuf; longjmp
TEXT runtime·gogo(SB), NOSPLIT, $0-8
  MOVQ buf+0(FP), BX   // gobuf, BX = buf
  MOVQ gobuf_g(BX), DX // DX = gp.sched.g
  MOVQ 0(DX), CX   // make sure g != nil
  JMP  gogo<>(SB)

TEXT gogo<>(SB), NOSPLIT, $0
  get_tls(CX)
  // 把要运行的 g 的指针放入线程本地存储，这样后面的代码就可以通过线程本地存储
  // 获取到当前正在执行的 goroutine 的 g 结构体对象，从而找到与之关联的 m 和 p
  MOVQ DX, g(CX)
  MOVQ DX, R14    // go 常用 R14 寄存器保存当前执行的 g
  MOVQ gobuf_sp(BX), SP // restore SP, 恢复 g 执行栈
  MOVQ gobuf_ret(BX), AX // AX = &buf.ret
  MOVQ gobuf_ctxt(BX), DX // DX = &buf.ctxt
  MOVQ gobuf_bp(BX), BP   // BP = &buf.bp
  // 清空 sched 的值，因为我们已把相关值放入CPU对应的寄存器了，不再需要，这样做可以少gc的工作量
  MOVQ $0, gobuf_sp(BX) // clear to help garbage collector
  MOVQ $0, gobuf_ret(BX)
  MOVQ $0, gobuf_ctxt(BX)
  MOVQ $0, gobuf_bp(BX)
  MOVQ gobuf_pc(BX), BX // BX = &buf.pc
  JMP  BX  // 跳转开始执行函数

执行 main

//go:linkname main_main main.main
func main_main()

// The main goroutine.
func main() {
    mp := getg().m
    // Max stack size is 1 GB on 64-bit, 250 MB on 32-bit.
    // Using decimal instead of binary GB and MB because
    // they look nicer in the stack overflow failure message.
    // 64 位系统上每个 goroutine 的栈最大可达 1G
    // 否则为 250MB
    if goarch.PtrSize == 8 {
      maxstacksize = 1000000000
    } else {
      maxstacksize = 250000000
    }
    
    // An upper limit for max stack size. Used to avoid random crashes
    // after calling SetMaxStack and trying to allocate a stack that is too big,
    // since stackalloc works with 32-bit sizes.
    maxstackceiling = 2 * maxstacksize
    
    // Allow newproc to start new Ms.
    // 标志主 Goroutine 已启动
    mainStarted = true
    ...
    //调用main.main函数
    fn := main_main // make an indirect call, as the linker doesn't know the address of the main package when laying down the runtime
    fn()
    ...
    exit(0)
    
    // 保护性代码，exit 出错也能强制进程退出
    for {
        var x *int32
        *x = 0
    }
}

Goroutine 调度

创建 Goroutine

从 Go 编译器代码可以更明确得知 go 关键字所执行的操作就是执行 runtime.newproc。

// src/cmd/compile/internal/ssagen/ssa.go
func InitConfig() {
    ...
    ir.Syms.Newproc = typecheck.LookupRuntimeFunc("newproc")
    ...
}

// Calls the function n using the specified call type.
// Returns the address of the return value (or nil if none).
func (s *state) call(n *ir.CallExpr, k callKind, returnResultAddr bool, deferExtra ir.Expr) *ssa.Value {
    ...
    // call target
    switch {
    ...
    case k == callGo:
      aux := ssa.StaticAuxCall(ir.Syms.Newproc, s.f.ABIDefault.ABIAnalyzeTypes(ACArgs, ACResults))
      call = s.newValue0A(ssa.OpStaticLECall, aux.LateExpansionResultType(), aux) // TODO paramResultInfo for Newproc
    ...
    }
}

调用 runtime.newproc，会先调用 runtime.newproc1 获取或创建 goroutine ，分以下几种情况：

随后 runtime.newproc1 还会对获取到的 goroutine 对象进行初始化，包括修改状态和变量，初始化 g.sched 结构用于调度等。

最后 runtime.newproc 会调用 runtime.runqput 函数将初始化好的 goroutine 进行入队，先把新创建的 goroutine 放入 P 的 runnext，如果有旧的 runnext，则尝试将旧的 runnext 放入 P 的本地队列，本地队列满则将旧的 runnext 放入全局队列。因为 runtime.newproc 调用 runtime.runqput 时，next 参数传 true。

其他的操作和创建入口 Goroutine 类似，只是执行的函数换成了实际所要执行的函数。前面没有提到的一个小细节是自 go1.18 后，编译器会将 go/defer 闭包调用标准化成普通函数调用方式。runtime.newproc1 原本需要将函数参数复制到调用栈上的操作被简化。

// normalizeGoDeferCall normalizes call into a normal function call
// with no arguments and no results, suitable for use in an OGO/ODEFER
// statement.
//
// For example, it normalizes:
//
//  f(x, y)
//
// into:
//
//  x1, y1 := x, y          // added to init
//  func() { f(x1, y1) }()  // result
func normalizeGoDeferCall(pos src.XPos, op ir.Op, call ir.Node, init *ir.Nodes) *ir.CallExpr

举个例子：

// main.go
package main

import "time"

func main() {
    aa, bb, cc := 1,2,3
    go func(a, b, c int) {
        println(a, b, c)
    }(aa,bb,cc)
}
 // GOOS=linux GOARCH=amd64 go build -gcflags=-S main.go
main.main STEXT size=89 args=0x0 locals=0x18 funcid=0x0 align=0x0
    0x0000 00000 (main.go:4)        TEXT    main.main(SB), ABIInternal, $24-0
    ...
    // 拷贝对象地址
    0x000e 00014 (main.go:6)        LEAQ    type:noalg.struct { F uintptr; X0 func(int, int, int); X1 int; X2 int; X3 int }(SB), AX
    0x0015 00021 (main.go:6)        PCDATA  $1, $0
    // 堆上分配内存初始化对象
    0x0015 00021 (main.go:6)        CALL    runtime.newobject(SB)
    // 复制 gowarp1 函数地址到 CX 赋值 struct.F
    0x001a 00026 (main.go:6)        LEAQ    main.main.gowrap1(SB), CX
    0x0021 00033 (main.go:6)        MOVQ    CX, (AX)
    // 复制 func1 函数地址到 CX, 赋值 struct.X0
    0x0024 00036 (main.go:6)        LEAQ    main.main.func1·f(SB), CX
    0x002b 00043 (main.go:6)        MOVQ    CX, 8(AX)
    // 分别将调用参数赋值 struct.X1, struct.X2, struct.X3
    0x002f 00047 (main.go:6)        MOVQ    $1, 16(AX)
    0x0037 00055 (main.go:6)        MOVQ    $2, 24(AX)
    0x003f 00063 (main.go:6)        MOVQ    $3, 32(AX)
    // 使用 struct 调用 runtime.newproc 函数
    0x0047 00071 (main.go:6)        CALL    runtime.newproc(SB)
    ...
        
main.main.gowrap1 STEXT size=141 args=0x0 locals=0x28 funcid=0x16 align=0x0
        ...
        
main.main.func1 STEXT size=135 args=0x18 locals=0x10 funcid=0x0 align=0x0
        ...
        
type:noalg.struct { F uintptr; X0 func(int, int, int); X1 int; X2 int; X3 int } SRODATA dupok size=200
        ...

执行调度

执行调度和首次调度 main Goroutine 类似，

Goroutine 退出

前面在描述创建入口 goroutine 的逻辑有说过，将 goroutine 栈顶返回地址指向了 goexit 函数，goroutine 在完成函数执行后就会跳转执行 goexit 函数

// The top-most function running on a goroutine
// returns to goexit+PCQuantum.
TEXT runtime·goexit(SB),NOSPLIT|TOPFRAME|NOFRAME,$0-0
    BYTE   $0x90  // NOP
    CALL   runtime·goexit1(SB)    // does not return
    // traceback from goexit1 must hit code range of goexit
    BYTE   $0x90  // NOP
func goexit1() {
  ...
  mcall(goexit0)
}

// goexit continuation on g0.
func goexit0(gp *g) {
  // 针对已执行完成的 goroutine 进行一些清理和销毁工作
  gdestroy(gp)
  // 再次调度，寻找可供执行的 goroutine
  schedule()
}
// func mcall(fn func(*g))
// Switch to m->g0's stack, call fn(g).
// Fn must never return. It should gogo(&g->sched)
// to keep running g.
// 主要作用是切换到 m.g0 执行 goexit0
TEXT runtime·mcall<ABIInternal>(SB), NOSPLIT, $0-8
  // 从 AX 取出参数的值放入DI寄存器，它是 funcval 对象的指针。
  // 此场景中 fn 是 goexit0 函数的地址
  MOVQ AX, DX // DX = fn

  // Save state in g->sched. The caller's SP and PC are restored by gogo to
  // resume execution in the caller's frame (implicit return). The caller's BP
  // is also restored to support frame pointer unwinding.
  MOVQ SP, BX // hide (SP) reads from vet
  MOVQ 8(BX), BX  // caller's PC
  MOVQ BX, (g_sched+gobuf_pc)(R14)
  LEAQ fn+0(FP), BX // caller's SP
  MOVQ BX, (g_sched+gobuf_sp)(R14)
  // Get the caller's frame pointer by dereferencing BP. Storing BP as it is
  // can cause a frame pointer cycle, see CL 476235.
  MOVQ (BP), BX // caller's BP
  MOVQ BX, (g_sched+gobuf_bp)(R14)

  // switch to m->g0 & its stack, call fn
  MOVQ g_m(R14), BX
  MOVQ m_g0(BX), SI // SI = g.m.g0
  CMPQ SI, R14  // if g == m->g0 call badmcall
  JNE  goodm
  JMP  runtime·badmcall(SB)
goodm:
  MOVQ R14, AX   // AX (and arg 0) = g
  MOVQ SI, R14   // g = g.m.g0
  get_tls(CX)   // Set G in TLS
  MOVQ R14, g(CX)
  MOVQ (g_sched+gobuf_sp)(R14), SP  // sp = g0.sched.sp
  PUSHQ  AX // open up space for fn's arg spill slot
  MOVQ 0(DX), R12
  CALL R12   // fn(g)
  POPQ AX
  JMP  runtime·badmcall2(SB)
  RET